I've got into quite a good pannier packing routine now. It's only on very unusual occassions that I forget something, and am wise enough now to have a spare pair of cufflinks and underwear in the office. In fact I reckon you could find a direct corrolation between days I've forgotten something and being out on the beer the night before! The sandwich shop down the road does a fantastic home-made soup and a big full roll for just £1.60 so there's never any need to bring lunch.
I'm dreading the day when I forget my towel though. I get showered in the ladies rest room, so 20 minutes of me drip drying might raise an alarm or two!