I don't mind people drafting me as long as they are not dangerously close and actually stop at the lights.
There is nothing worse than them nob heads who use you for a free ride, sale past at a red forcing you to overtake again, only to find them sitting on your wheel a few seconds later! Cheeky B*$%^!s
There is this one chap I enjoy riding with, we take turns up front yet never even acknowledge eachother - Thats alright

Other than that I tend not to draft people.
You need to be confident in the persons ability, if its a stranger (like you pointed out OP) its not a good idea to draft!
He was very much in the wrong. Especially giving you that cheek when he is looking for a favor to you. I dont see what his problem was, I'd have had a go at him too ducky!