I don't think I've ever had anyone on the road accuse me of not paying road tax. In my experience drivers don't hang around very long. At most all I hear is "Wankeeeeerrr...." as they drive past in their company liveried vehicles.
I have had the pleasure of people at work or friends being a bit put out that cyclists don't pay tax. When it comes down to it, most people are like Lizban seems to be suggesting - they don't actually care why they are taxed, and what the money is used for - they just care that drivers are taxed, and cyclists are not.
For many people driving is not seen as an option - its something "everyone" has to do, and to those people it seems unfair that they pay exorbitant rates to be on the road whilst others seem to be getting a free ride. And there's not a lot you can actually say to people to dissuade them of this opinion, especially when they get their arguments all in a muddle.
Lizban's argument about red diesel reminds me of the time when I was berated for not having to pay the M6 Toll as a cyclist, despite the fact that, as a cyclist, I couldn't use it. When I pointed this out they brought up red diesel, and how they saw a tractor on it which wasn't paying fuel duty and therefore cyclists should subsidise the road... I mean WTF?
At the end of the day, most driver arguments can be responded to thus:-
You don't pay road tax. I pay council tax, therefore I pay for the road.
But you don't pay car tax. Because I'm not driving a car.
But you don't pay fuel tax. Because I'm not using fuel.
You don't have insurance. Yes, I do.
Gerroff the road. No.
When all else fails, do as Jacko suggests and play deaf.