I'd do it up, you said that going downhill was scary, to my mind, just replacing the brakes with something a bit more powerful, maybe the latest ultegra, and some really good brake pads would solve your problem, and probably cost about £100. You didn't say what components your bike is currently fitted with, but I'd think something like the below would help.
https://www.merlincycles.com/shimano-ultegra-6800-brake-calipers-pair-62941.html
Add some koolstop salmons (other people will suggest other pads) and you'll probably find that theres a big difference in braking performance.
For the age of your bike, I wouldn't have thought it would be necessary to upgrade the whole bike, just update those parts where you're looking for an improvement in performance, in this case brakes. You can then mix and match from cheaper/more expensive groupsets where you want to prioritise investment.
https://www.merlincycles.com/shimano-ultegra-6800-brake-calipers-pair-62941.html
Add some koolstop salmons (other people will suggest other pads) and you'll probably find that theres a big difference in braking performance.
For the age of your bike, I wouldn't have thought it would be necessary to upgrade the whole bike, just update those parts where you're looking for an improvement in performance, in this case brakes. You can then mix and match from cheaper/more expensive groupsets where you want to prioritise investment.