See, you're forgetting to take into account the fact that the earth is turning.
Ride in, into the wind, by home time the earth has revolved and you're actually riding the same direction again.
Or something.
I do know that even if you get a tailwind, about once a century, it's never as strong as the wind going the other way.
I was once riding back towards work, when I saw a colleague riding home coming towards me, on Stockton Lane. We passed each other with gritted teeth greetings, as we were both riding into a headwind.