I get this too.People who say “That looks like hard work” when I’m out enjoying riding my bike.
Clearly most people have a dysfunctional relationship with exercise. Plus they must have a poorly calibrated index for what hard exercise really is.
I used to get more annoyed when I picked up my kids from kindergarten or school with the Bakfiets or Xtracycle, and people would say "My word, you've got it good haven't you?" to my children, while stuffing their own kids into the back of a car.