Look, if that can help make my point, I just devised a story. Suppose you're an honest law-abiding citizen. Suppose you're black. Suppose you live in a country where there is some strong anti-black racism. Sure, you know that the majority of the population is made of normal rational people. However, the racist minority is really making your life hard, in fact they're making it dangerous because they'll go to extreme and irrational levels of violence against you simply because you're black. Now, you have very well noticed, from newspapers, in the media, in fact from everywhere (those racists are also very vociferous) that those racists employ any incident involving a black person as a justification for their racist views (things along the line "Did you hear about that guy who robbed this poor old lady? Yes, he was black. And people call me racist?"). And you infer that those criminals (only the black portion of the criminal population, the rest doesn't matter) are partly, in fact greatly, responsible for all this racism that affect your daily life, because it reinforces the racists views of racist people. So, do you go and try to convince every single black person in this country to stop committing any crime, as an attempt to eliminate racism? Or do you feel that this is exactly what racism is about (the non-racist doesn't give a shoot that the aforementioned criminal is black, and doesn't relate you to this black criminal, in fact he doens't give a shoot that yourself, the criminal, or anyone else is black or not), and what you want is indeed to fight racism, by fighting racists themselves?
Does that story sound sadly true? Now replace black person with cyclist and racist person with driver holding negative views of cyclists, read again, and draw your own conclusions.