I've been riding through the winter since I was 6, and the first flake of snow had me heading for the shed and hailing out the Trance. At work its great, and in bad snow I've got to places no one else could, one one occasion sitting with a casualty alone for several hours because no one (except one of my students, a PCSO who later arrived to help) could get to him.
In 37 years of cycling I've not had that many Offs, and most of them in the dry. Trust me, a face plant in August hurts as much as in February.
And on top of it all the poor surfaces, weather, poor visibility etc can improve your fitness, core strength, control skills, anticipation... your not doggedly keeping your fitness going on an indoor trainer, you're missing a trick.
I appreciate they can be very useful tools for stuff like heart-rate based exercises etc, but as a substitute for actually riding... nah.
