As almost everyone has said, wear what you want; anyone who looks down on you for wearing regular clothes isn’t the sort of person whose views should concern you.
Ironically, when younger and slimmer I would have been way too self-conscious to wear lycra, and often cycled long-ish distances in regular casual clothes (I learned not to wear denims after cycling in jeans from Edinburgh to Glasgow - a torture device after about 10 miles).
But, wearing what you want shouldn’t exclude specialist cycling clothing. Now that I’m older and fatter I don’t care what I look like on the bike - and cycling clothes are more comfortable for longer trips. You can also get more understated designs if you are worried about the occasionally garish appearance of cycling jerseys.
There is an element of snobbery about what you wear in some sections of the cycling world, but there also appears to be an inverse snobbery as well. Basically, damned if you do and damned if you don’t.