What a road bike
can do and what one
should do are two different things!
I bought a Bianchi about 10 years ago; quite an expensive bike - £1,000 for the frame/forks and about £1,000 more for the other parts. I decided to take it up to the Yorkshire Dales for its maiden ride. The intention was to do a big loop over Fleet Moss to Hawes then back via Ribblehead to Horton-in-Ribblesdale but by the time I got to the top end of Wharfedale my back was killing me. I didn't have a map with me, but I'd got it in my head that I could follow Langstrothsdale Chase round to H-i-R so I headed up there instead. It was a lovely little singletrack road and I was enjoying my change of plan until... the road started going uphill and turned into a bridleway
I couldn't face going back, so I pressed on but felt that I was being terribly cruel to my pristine new racing bike. The ride became a bit of a nightmare. Slick 23C tyres don't grip mud and gravel. Lightweight racing wheels don't like bouncing over boulders. Lightweight tubing doesn't like having rocks bounced off it. By the time I finally got to Horton, my tyres were cut up, my frame was scratched and my nerves were shredded. I'd had a couple of very near misses on descents.
So, you
can ride road bikes off-road, but I wouldn't recommend it. If you get a bike with enough clearance for bigger tyres, that would be different. I'd want at least 28s but preferably something bigger. Some sort of tread as well - the tyres wouldn't roll as well as slicks on tarmac, but at least you'd have some grip off-road. I saw a heavily laden touring bike being ridden up a steep local bridleway which was covered in rocks. It looked hard work, but the rider managed it. He had something like 35s on his bike.
If you are going to do the road rides on different days to the offroad rides, why not buy a road bike and have two identical pairs of wheels, one with narrow slick tyres for road, the other with bigger knobblier tyres for offroad? (You want identical wheels so that you don't have to adjust your brakes every time you swap the wheels over - I made the mistake of getting two different rim types and the brakes didn't work when I swapped wheels

)