Must have been about 8 years ago I was sent to A&E with an unsual skin infection in my hands after grating them on tarmac; I've worn fingerless gloves since, or something warmer in winter.
Thinking about it, my clothing choices are usually based on bad experiences. Eg my wrecked "undercarriage" from touring in jeans means I now wear padded cycling shorts for long rides. <crosses legs>
There's also the issue that cycling can cause clothes to get dusty/wet/muddy/mucky/oily/sweaty. Do you want that on your best suit? Better to wear specific cycling clothes surely, although you might also be able to avoid the problem by using a bike with mudguards, chain guard, etc.
Sweat can be avoided by reducing effort.
Personally I stick to office clothes for commutes up to about 4 miles each way.
Jeans for rides to the shops, dentist, family rides, etc.
But if I'm doing a solo ride "for fitness", or for longer commutes, then I'll dress for comfort and practicality, and that means lycra. Plus it scares my colleagues.