I think it depends on the 'other bike' you're comparing it to.
Most (i.e. >90%) of what's stopping you from moving forwards on any bike is air resistance. The more you can do to cut more cleanly through the air, the better.
Bromptons by nature are very much sit up and beg compared to a modern road bike. So that's the main reason why they're slower - it's aero drag, rather than wheel size or anything else. Whenever I've done the same route on my Brompton and Road bike, I've noticed around a 2-3mph difference on the Brompton.
That said, in the summer I did the same 19 mile route on my Brompton as on my Road bike and my average speed was only one mile an hour slower. But then the wind speed on the road bike was 15mph while it was only 5mph on the Brompton and I did take in a detour on the road bike to take in a different hill.
Top speed was 34.9 on the road bike and 33.3 on the Brompton. And I know that on my normal commute, top speed is almost identical at 44mph. But then, I'm not pedalling on that downhill and trying to get as aero as possible.
Brompton:
Giant TCR Advanced:
It was surprisingly similar over that relatively short distance. But where this will really start to take it's toll is on longer rides. You're putting in the same effort and going slower, so therefore you tire more quickly because it all takes longer to complete. Or you have to work harder to complete the same route in the same time.
Either way, the end result is that you're more knackered.
I've not been out on my MTB in years, but went out with the missus at the weekend and it felt like a right slog. She's got an electric bike so that she can come out with me on my road bike and not hold me up (too much). But on the MTB, I could not keep up with her at all on the tarmac sections. Big fat tyres and upright riding position certainly didn't help.