Any fandom who manages to raise 2.8 million dollars to help fund research for incurable diseases can't be all bad, right?
Pretty much all fandoms are annoying. Most of them contain some people who aren't annoying. If you want to cosplay in that fandom and you don't like annoying people, seek out the people who aren't annoying. They exist; I know many of them. I am sure there are annoying people in small fandoms, too, just as there are quality, non-annoying people.
The two major fandoms I have cosplayed with others are the ones I've seen brought up time and time again in this thread. Because they are large fandoms popular with young people, the percentage of annoying fans one is exposed to at any one time is going to be higher. For example, I am sure if I went to a Metallica concert in the USA, there would be a lot more annoying people there than if I went to a Harmaja concert in the USA, but there would still probably be annoying people there.
Personally, whatever fandom it is that yells "Marco Polo" is the one I find the most annoying. So the world explorers fandom?