A mate of mine had the same problem on the Manchester 100, typically, the one year I was actually fit enough to do a decent time.
I dragged him round the first 55 miles to the 'halfway' stop and by then his legs were shot. We spotted the problem just before setting off again and realised that he must have knocked his rear brake when putting the bike in his car. We still managed to do the 100 miles in less than 6 hours so I reckon a sub-5 hour time would have been on if not for that.
Despite that experience, I managed to do exactly the same thing on
The Red Rose Ride a few years back. That is a local hilly 200 km audax ride which felt so hard that I eventually asked my riding companion if my back brake was rubbing. He told me that it wasn't so I assumed that I was just not fit enough. When we got to the car park at the end of the event I dismounted and immediately discovered that my bike was really hard to push over to the car. I'd ridden about 128 miles with the brake rubbing! I asked my fellow rider how come he said it
wasn't rubbing and he realised that he'd only looked at the right side block and it was the left that was the problem ...
On one of my forum rides I was going even slower than usual but I finally realised that something was seriously wrong when I struggled to do 15 mph downhill on a road that I could normally freewheel down at 25+ mph. That turned out to be due to dried mud packed under my rear mudguard and rubbing on the tyre.
It's amazing how much difference that amount of extra friction makes.
