A crosswind can move me on a 40kg Brox. It sits lower, weighs more and has the extra stability of four wheels.It still makes a difference though. My lightest bike is just under 25 lbs, the heaviest one just over 40. All have small diameter steel frame tubes and conventional wheels so the aerodynamics are not greatly different. If I get hit by a gusty cross wind on my 40 lb 3-speed it doesn't have much effect apart from being unpleasant. The same sort of gust is enough to actually blow me noticeably off course on a light bike and make me correct the steering. The overall difference in weight if you include me is small, but when riding a heavy bike the extra weight is added to the extremities not concentrated in the middle as with the rider.
At times, more than I'd be on a bike. The constant in both being the rider. Weight and side cross section. The rider adapts to the conditions, once they get used to the cycle their riding.