It's all about the gear. I know it seems like an expense but if you get warm leggings, a cosy jacket, a decent pair of gloves and overshoes if you use clipless pedals, then you're on to a winner. I doesn't matter what the weather says, if you're prepared then you'll be fine.
It's the leaves on the corners of the road that scare me