This thread is useless without photos!
Seriously though, it's a pain in the 'arris and needs to be sorted. In 99% of cases you can look at a girl (or a bloke, come to that) in lycra and think "hmm, looks fit" and that's as far as it goes. Unfortunately though, there are some sad and lonely people around who aren't quite in touch with reality and try to take it further. I was once being harrassed at work by an older woman who kept groping me and fondling, it was an irritant but I was never going to complain officially about it. One day though she grabbed me as I was in a hurry and stressed and I turned round and snapped at her, a genuine gesture of annoyance. She's never done it again. Maybe you just need to lose your temper with this bloke?