I wonder what percentage of cycle commuters do 50 mile round trips 5 days a week. I'm guessing it is single digit %'s
I suspect you're right. It requires a degree of bike fitness that doesn't just happen. That is not the same as saying that it's out of reach for the majority though.
The point is, it's not due to lack of time that it's in the single digits.