The problem with passing someone like this is that you have to make it stick. In other words, to avoid the world's greatest ignominy, (being passed by someone you have just passed), you have to ride like stink whilst appearing nonchalant, and.........and.....without checking over your shoulder to see if he is coming back at you. So you can end up riding flat out for 2 or 3 miles, not knowing whether the guy is drafting you, is half a mile back, or has turned off.
The other approach is to only pass on hills, and then to wait at the top of the hill and get chatting when he catches you up. Maybe ride together for a while. That way you don't have to worry about him attempting to repass you. That's my Sunday morning preferred option.