It depends a lot of what kind of riding you're doing, too. If you're riding on safe roads - as in there's nothing majorly slippery on the surface - you can push on and keep your body producing warmth, so you maintain a fairly steady temperature. If you're riding a mixture of safe and slippery roads, you get times when you're pushing on and times when you're having to take it slow, so you're liable to be too hot in one situation or too cold in the other.
My worst experience was mountain biking on Friday. I climbed over 800m, and I was pretty warm on the climb in just a thin baselayer and windstopper jacket (undone), mitts and nothing warm on my neck and head. By the time I'd done the descent into a northerly wind, I was so cold I literally couldn't feel my fingers and my stomach muscles were cramping. I really needed to carry a warm jersey, buff, hat and winter gloves to put on for the descent.
So, from my experience, the only option if you're doing varied rides in cold conditions is to be prepared to stop often and add or shed layers.