I'm sorry my post about the plastic bag wasn't clear - I didn't express myself too well.
I meant to suggest having a plastic bag to take in & out of the pannier, with all your stuff in.
The best ones are probably like those you pay 50p for at supermarkets, for longevity. Not neccessarily for waterproofing, just for sheer convenience.
I know what you mean about having extra weight on the bike. I used to have a pannier on for winter, then a rackbag, and having taken off the rack, I now simply have an ancient sort of cordura belt-bag with a shoulder strap which I use round my waist like a belt. It's not quite a bum-bag but that would probably be the closest thing to it. It holds wallet, phone, multi-tool and packed lunch. Waterproofs get stuffed into my back jersey pocket. If I need to take clothing into work I use a backpack.