Understandable you should be annoyed and tempted to react, but best to not let it get personal and just let it go; anyone who behaves like that is obviously a half-wit and whatever happens to them they won't be 'cured' of their half-wittery. A few hours later it won't matter half as much to you - that's what I have found anyway, after being verbally abused by d***head car drivers.
You just don't know who you are dealing with, but regardless of how good you are at fighting, I would only ever do so as a last resort to protect myself or someone else, even if I was good at fighting, and even if I liked it.
When I hear of those cases where people have been killed in a momentary fracas, often as the result of a single punch, it scares me silly. I can reel off at least 4 cases near where I live that have happened in the last few years. 2 involved good mates who fell out whilst drunk - both single punches, another involved a dispute at a cash point and another involved a bloke getting a few thumps after intervening when some girls seemed to be getting harassed by a couple of lads leaving a night club. In the last case a young bloke who had a very good professional career in front of him is now doing life imprisonment for murder and 2 families have been devastated. Might sound dramatic but that really is the potential of getting involved in a fight. Unless you have to fight it is much better to be an alive and free chicken than a dead or imprisoned hero.