I see this again and again at time trials, many of which are in the East of England where it is flatter but usually windier. Essentially everyone is in a good aero position on the tri-bars but on a windy day the lighter riders are slower compared to the bigger riders. I have some fairly good proof of this, last year in a 25 mile TT near Wisbech in the Fens, the wind was blowing hard in one direction on the course which involved 2 laps. My better half who is very light but has a good power to weight ratio was as quick as anyone with the wind behind, however, into the wind [and it was blowing hard] she lost a lot of time to to the men and some to a couple of bigger women. We know what her comparative times are to those she races regularly against and it was interesting to see, if disheartening the effect of the wind.
Racing a bike or just riding quickly is mostly about power to weight ratios but there is a tipping point when it is negated due to the strength of the wind. Light men will deal with it better than light women because their power to weight ratios are even higher.