After breaking my spine, I stuck to canals and bridleways for a long time, then started back on the road down at the caravan we had in North Wales as there were many quiet lanes. I got back to commuting 3 years ago but still use as much off road as I can (less so in evenings as my route has scallies on it when they can get out of bed).
My latest big accident was just unlucky, slow fall on black ice despite having studded tyres.
I was back once my consultant was happy with the bone healing (three months due to misdiagnosis).
I'll always say, mix up your cycling. If you just have road bikes, then I'd suggest an MTB or gravel bike depending on terrain locally.