Being larger myself, I've come to realise I look more daft trying to wear baggy clothes than the more form fitting lycra.
In fact, baggy clothes tend to make me look bigger, wider and chunkier than I actually am.
I still wear shorts over the lycra padded shorts, but everyone whose seen me tells me to lose the shorts I wear over the top because they look silly and I look perfectly okay in just the lycra padded shorts.
My mind however isn't ready to be that wild yet.
Both jersey and shorts are extremely more comfortable too.