Too many folk in life are bothered with what others do/wear rather than live and let live. If you want to wear your polka dot jersey, and can ignore the odd eejit who has nothing better to do than tell you they object to that, then wear it. As others have said, people wear football & rugby tops without playing for the team concerned and no-one questions that. At least with cycling clobber it's usually worn while actually cycling (although I did see a fella walking a dog in a cycling jersey and jeans the other day and, contradicting my live and let live mantra, personally that struck me as simply wrong).
Seems to be snobbery and unwritten 'rule books' with many aspects of life that are simply recreation - I've experienced them in football, fishing and cycling. Some are well intended, some preserve tradition and decent manners etc., but many are just folk getting above their station. Too much time spent having to comply with what others want - cycling is your time to wear what you want.
More things to worry about in life than judging someone by the top they wear cycling (like my skinny hairy legs poking out from the lycra shorts :-)