You'd notice a difference for sure but it depends on how you ride, and what you carry, as to how happy you'd be with those differences. I started on one bike with 35's, went to 28's, then 23's and back up to 32's. My weekend bike currently has 28s(mainly because they were on offer I'd planned on 32s) but I'd go bigger if I was carrying luggage etc. The commuting bike has M+ 35's and, if they ever wear out, then I'll probably go up to 42's for that bike.
If you take on a lengthy commute then it's worth thinking about the roads/tracks you'll use, if you'll have panniers, backpack, mudguards, etc. When I commuted 40 miles daily my bike could feel a bit sluggish in the mornings, or at the start of the week. By the way home, or the end of the week, I was very glad of the Brooks saddle, bigger tyres, plusher ride and puncture resistance offered by the M+. For me it's a no brainer, comfort over speed and, to be honest, I really don't think you sacrifice as much speed as you expect. Or maybe my top speeds weren't what I thought they could be