Did you speak a fraction of a second before you came alongside him, and so visible in his peripheral vision? I have a local rider, faster than me who does this in a very loud voice with 'GOOD MORNING!' right in my ear every blasted time he passes me. If I'm day dreaming he fair shocks me as he does this. I haven't said it yet, but one day that rude grumpy rider will be me.
If I see him coming in my mirror I'm ready for this and not shocked, and I always speak to other riders but not like this clown!