I genuinely don't mind going out when it's cold. It's wet and windy I'm not quite so keen on. Says she who cycles out in the fens where it's *always* windy...
Although going out when it's not so warm does mean getting the clothing right. Often, I'm out at dusk at this time if year, because that's usually when I can squeeze a ride in, so tend I go out a bit overdressed. Reason for this is that out here, the temperature drops like a stone once the sun goes down, and there's nothing quite so miserable as being cold on a bike.
Best advice is layers, layers, layers... Just like an onion.
And keep the extremities (head, hands, feet) warm.