Why do some people dislike all Catholics?

I'm not saying that ALL people are prejudiced when it comes to Catholics, but why are some people? If one priest turns out to be a child molester, that doesn't mean that you should judge the all the Catholics. All religions have some bad people, and the Catholic Church, being so large, is bound to have some also. Why do some people judge us based on what they heard about a priest on the news? Most of those people don't even know anything about Catholicism.