I'd say it's because a lot of people start out as "blokes who ride a bike" then it gradually morphs into an obsessive way of life that takes over too much free time, prevents you from eating and drinking any "bad stuff" you want, and becomes something that is eventually resented because it controls you not the other way round.
Bloke starts out on a fairly basic and inexpensive machine, and it gets him from A to B and is a relaxing and harmless hobby. Then starts to read media reviews of fancy equipment, interact with people on forums who really think that £5k is a perfectly normal price for a push bike, looks at the weight of everything down to the last gram, has to have a separate wardrobe of cycling-specific clothing, gets a fancy cycle computer, signs up for Strava, starts obsessing about average speeds and achieving ever faster times over their regular routes. Forces themselves to go out on "training rides" even though the weather might be horrible, and they'd actually much rather spend the evening in front of the TV or down the pub. Feels guilty if they don't ride as hard and as far as possible as often as possible, because they might be missing out on a potential higher level of performance.
What starts out as a bit of fun becomes something that absorbs all of someone's spare time and all their spare money. I would actually consider it a form of addiction in some cases, especially when it gets to the point of people admitting to being bad-tempered and grumpy because other things in their lives (like work and domestic responsibilities) have conspired against them to prevent them riding for several whole days!.
Never lose sight of the fact that it isn't all about average speeds and longest distances and PB's and KOM's, it's just about jumping on a bike and going for a ride. Even if it's a tatty old worthless heap of a heavy steel bike and you are wearing jeans and a pair of builder's boots. As a casual utility and recreational rider, I can take it or leave it. Go for a ride, or go out for a load of beer and a doner kebab instead. If I choose the latter, I don't feel guilty about it and I don't feel the need to "make amends" by riding twice as hard or as far the next time. I own my cycling hobby, it doesn't own me - and because of that I don't ever feel forced to do it when I'm not in the mood or resent the amount of time it demands.