Are you wearing a woolie beanie?
They are useless. Hold too much sweat in and that goes cold, and dont block the wind.
Have a look at skull caps from gore, specialized and assos. Not cheap but worth it for regular cold rides.
Edit: some cycling specific woolie onesare suposed to be decent, I meant the £2 woolie ones off the market.
Edit 2: merry Christmas.