I agree, but I think you can make it work fairly and well, at least I think I did. Yes, you could argue being out for a ride early on a morning is taking time away from the family, but if it recharges your batteries (and regardless of having a family, IMHO you are still an individual person, and I think it's a good idea to cherish and nourish your own identity -of course family comes first, but I'm trying to say that keeping yourself mentally and physically healthy for you and your family is also a very, very good idea) I think that can be very good..... within reason*. Also, if like me, you took it in turns to sleep in, cycling in the morning when its your turn to sleep in isn't effectively taking anytime away from the family as you'd be asleep anyway. And of course, as a parent, your family should come first.
*completely agree with you about recovery, I'd be peeved and I'm not saying that you should be doing long bike rides all the time and then effectively being zonked for your family the next or same day, which isn't a smart thing, and is a detriment to your family!
Being completely blunt, is the problem the children or the wife?
Are you not going out for a ride because you want to spend the time with your children, or would the wife prefer you to be at home?
The obvious answer is to go before they wake or after they're in bed, but either way it's sacrificing time with your family.
Personally I wouldn't bedgrudge my husband a few rides a week, but I'd probably start to get a bit peeved if he was spending every saturday out on a ride and sunday recovering.