It's not too much of an issue about the money, as indeed, you could (and would) be charged. It's about the numbers. The numbers that follow regarding drivers and cities are taken from a quick Google, so I give no guarantee on their accuracy!
There are 30 million drivers in the UK, approximately. For the sake of simplicity, let's pretend they are evenly spread out in the year they passed their test, which means there are 6 million drivers that need to re sit their five yearly test this year.
There are 50 cities in the UK. Let's assume one testing centre per city. I don't think this is unreasonable - certainly there wasn't a testing centre in the town I grew up in and I had to go to the nearest city instead. Again, assuming that the drivers are spread out evenly across the UK, this means each centre has to process 120,000 drivers each year.
That's 328 drivers every day. That ignores any retests (if allowed!).
Assuming the test would take roughly an hour (call it 40 mins on the road and 20 mins admin), and that's eating centre work a twelve hour day, that means you still have to give 27 tests every hour - so you'd need about 27 trained examiners working twelve hour days, seven days a week!
Even if you up the number of test centres to 300 in the UK (three per city, and an equal number around the rest of the uk), and have examiners working twelve hour nonstop days, seven days a week, you still need five examiners doing these hours at every single centre to process the numbers!
It's just too much to process.
However, if you make it a necessity to resist the driving test if you are involved in an accident and the insurance companies (or police) deem you to be at fault, I suspect that would make the numbers a bit more realistic... This would include damage only, no one else involved accidents like the one the OP saw - as long as he had to claim on his insurance ofc.