I think typical driver aggression varies by the time.. in rush hour, everyone is pissed at traffic, including you (especially if you are going slower than the cars). It makes perfect sense that the possibility of not getting to work on time is annoying (not saying it's rational to assume one cyclist will be able to make you late, but it's a minor nuisance that gets blown up if you're really trying to get something done).
I mean, don't you get annoyed on your bike when there are huge queues of automobiles but no room to filter?
Best way to avoid the situation is to ride nearer the kerb unless you are doing the same speed as the cars (or you are approaching a junction, though everyone slows down at junctions anyway so that's kind of the same thing). This is why I find main roads safer during rush hour than side streets, there are so many cars and so many junctions that no one can do more than 20mph or so.