i think it depends what is it. for instance i've got a road bike with XTR pedals and Ultegra group set, and i've got a cross bike with normal shimano M520 pedals and a mix of tiagra and 105 groupset.
i don't notice any difference in the pedals, but i notice difference in the gear change. The tiagra is clumpier.
the road bike is a sub-kilogram carbon frame, the cross bike is heavier aluminium and when i first had the cross bike, it had skinny road tyres, and my times on both bike are pretty much the same. The only difference is the carbon is better for climbing, but then more skittish on the descents because of the lack of weight, and i feel the carbon shift more when a side wind catches it, which can be a bit hairy at times, so the aluminium cross bike is a "safer" ride i think, but doesn't seem to be any slower. I enjoy hill climbs less on the aluminium, but safer on the downhill descents, and even though i gain time uphill on the carbon, the average speed stays the same (so i must get down faster on the aluminium?) which makes we wonder if we're all being sucked into a marketing ploy to buy more expensive bikes.
horses for courses i think. i use the cross bike for commuting/winter and the carbon for "proper rides". Buy the cheap end stuff for commuting and the save the nice expensive stuff for sportives, racing and long days out with friends when you need smoother lighter riding to keep up!