I agree with you there.
Last month I had a guy try to overtake me even though I was doing 30mph+ drafting a van. He then beeped me when behind, and finally overtook me in a 20 limit with double white lines.
I was angry to signaled for him to pull over, which he didnt.
Its interesting how people react differently depending on what happens. If I shout something at someone, and they hear, they will probably have a go at me. However, if I just gesture, and they see it, they normally just ignore it. And if someone tries to knock me off, they will run away.