Some of it is about the bike.
I'm certain that if I was on a Dogma and Bradley Wiggins was on a 3 speed shopper he'd thrash me, but if I was on a Dogma I'd be quicker than I would be on a shopper.
Having said that, I can't imagine a top end 3 year old frame being significantly slower than a new frame, but as already mentioned, things like worn or badly maintained hubs, chains and mechs might make a difference. Riding with 'sticky' hubs is definitely noticeable.