It depends on each persons individual circumstances and attitude to risk.
For me, it's CO2 every time, as it's more convenient and cheaper than buying a mini pump. I've only had three punctures out on the road in the last 12,000 miles, so I only carry an inflator, one cartridge, one inner tube, one of those tiny packs of Park patches (about the size of a postage stamp!), and a tiny schrader/presta converter thingy so I could use a pump at a garage in an emergency. The whole lot fits into a zip lock bag about the size of a fag packet, so it easily fits into a jersey pocket.
Obviously, there is a risk I might not be able fix a puncture if I have more than one, but I think the risk is acceptably small. Realistically, there is more risk of something else going wrong like a broken spoke/rim or a slashed tyre. I take a phone and some cash with me, so I'll probably be able to get home somehow!
I've seen more problems caused by over furious pumping tearing the valve out of the tube than I've seen with CO2. It's not easy to get to a high enough pressure for narrow tyres with a mini pump, and you need the right pump. Some are useless.
If I had a pump attached to the bike, it would probably get destroyed by the salt over winter anyway. I don't understand the cost argument - the inflator heads are a couple of quid from
Wiggle and cartridges are about a pound each.