Paul, I have the benefit of having met you and I think you are of a similar age and at a similar stage in your life to me. I also have an interesting and enjoyable job which pays well and doesn't stress me too much. Two years ago my immediate colleague, who was 10 years younger and more impatient as well as very materialistic and cash driven, talked himself into a position with a competitor which, on the face of it, looks enviable. However he has put himself into a situation where he will need to perform extremely well in order to deliver what he promised at the interview and we his former colleagues are watching with interest to see what happens.
At our stage in life I believe we need to be taking care of our mental and physical health a lot better than we might have done in our 20s and 30s. If cash isn't an issue and if you have the stimulation of your own business, why stick your head above the parapet? It sounds complacent but as others have written above, it's all about achieving the right work/leisure balance. How would your family feel if your stress levels and absences suddenly increased?