I've never had anyone attempt to steal anything from my panniers - but then I use the Torpeak quicktrak system, so taking the pannier off is so easy I tend to take it with me whenever I dismount.
That said, I have left my wallet, car keys, recent purchases in the pannier, and not had any problems either. I may just be lucky, have misplaced faith, live in a good area... dunno really. My pet theory is that folk, on the whole, are good people. That tends to steer me straight.
Someone will post that will prove me wrong of course!