If it means the end of the "me time" then no.
Cycling is the only thing I do which is mine personally and I rode a lot alone as well as sometimes with friends. I have been married for 13 years now and am deliriously happy with the relationship, my family and our life in general.
We all need a little time alone doing something we enjoy for ourselves which in my case is usually early morning rides on a weekend and directly after work for an hour a couple of times a week. This means it has no impact on the rest of our lives.