I was an IT contractor for a few years. I used to get strange looks from people when I'd tell them I was giving up a day's pay to go beach fishing.
Still, at the time I was single, all my bills were paid up and I had a bit of money in the bank. My financial situation has changed drastically since then (and that's not just down to being married witrh kids) but I don't regret any of the days I took off. I've always worked to live - not the other way around.
As my eldest says - YOLO.