"One bad apple spoils the whole bunch."—Commonly used idiom when it comes to a person's perception of a group upon seeing its Vocal Minority.
When it comes to fans of certain things, there are many people who are sane, down-to-earth individuals; individuals who understand that if they talk about a subject, and the other person's not interested, they'll stop talking about it. These people know how to keep their hobbies under control; how to differentiate between reality and their interests.
Unfortunately, these people never show up on the news. That's because Weird Is Interesting - Like any other program on television, the news has to be interesting or else people won't watch it. (Not to mention on how the media can be notorious in exploiting the trouble makers and overall embarrassments to their hobby/profession every chance they get, and the occasional Manipulative Editing. While any good people like that get a mention infrequently, if ever.) Why talk to an individual who works in an office, who occasionally picks up a copy of Spider-Man, when you can talk to the 300-pound guy who spent $7000 on his exact replica of a Scarecrow costume? Why speak with one of the female fans; after all, girls who read comics? Those don't exist, right?
The problem is that because only the weirdos get interviewed, this causes non-fans to believe they're all like that; that all comic fans are clones of the Comic Book Guy from The Simpsons, for example. This is also the main reason why people would hate something just because of its fanbase. It's like the old saying: "The squeaky wheel gets the grease". Not to mention on how the more Vocal a Vocal Minority can be, the more Demonization it gets, regardless of whether the other side is just as bad if not worse. So be careful if you demonize them because in the end, if you're not careful, all you'll end up doing is giving them a reason to deem themselves a "victim". (Regardless of whether or not they actually deserve it.)
Not just limited to Comic Books, this phenomenon extends to other hobbyists, other groups and races as well. Although there can be odd immunities: compare how your average American treats a guy in a Spock costume versus how he treats a guy with his face painted the team colors.
Sadly, there is a tendency for many of the Vocal Minority to seek position of power in their groups, making matters much worse. Although it makes sense: the guy who cares the most about a topic is probably going to get a leadership position because he knows the most and is around a lot.