What Banjo and yello say is sensible - and unless you're up against the time limit due to having loads of punctures/mechanicals hanging around for a few minutes for a stamp won't really have much of an impact. I've ridden quite a few 100s and the longest I've had to wait to have my card stamped is about 10 minutes - the control was only about 20k from the start so everyone arrived at a similar time. There haven't been that many people on the 200s I've ridden, so so far it's never been an issue.
It's a good opportunity to have a stretch and a chat, it's all part of the ride experience and handy to be on nodding terms with other riders just in case you do need a bit of assistance at some point.
However, if you do decide not to bother with the stamps, please spare a thought for the organiser and give them a courtesy call to say you've finished, it spares them from wondering what happened to you (some will just assume you've packed in, others will worry a little more!).
Just out of interest, where did you read about long queues? I'm aware that there can be long queues on LEL and PBP because there are so many people on the rides and Arrivee magazine mention those, but didn't think it was a common problem on shorter rides.