I was once stuck for 30 mins behind a tractor all because the car immediately behind wouldn't overtake.
The angry driver had probably one or two similar bad memories - personally I would have done what you did.
It's always been my personal pet theory that the large queues behind slow moving vehicles aren't really caused by the slow moving vehicle itself but people who either have no intention of overtaking or sit so close behind that they can't even see if it's safe to overtake and there is then no room for anyone who does want to move his way up through the queue of cars to go.