Greed. And poor/no regulation of insurance companies.
I run a team of 20, most of us employed precisely because regulation of insurance companies is tight, and the industry takes it seriously. Have a look at Consumer Duty and the FCA's clampdown on Direct Line.
I'm pretty sure their actuarial statistics don't go down to street level, but only down to the out code (First 4 characters of the postcode). And I think they take into account all types of crime.
And more importantly than just crime, they also take into account the general claim history of the area, including the vehicle demographics of those claims. So if your area has a lot of young drivers, making more claims in the low insurance group cars, premiums for those may go up, while higher spec cars may go down relative to other areas.
Address point pricing (house level) has been around for at least the last 15 years. And your second point isn't correct. The models used since the late 1990s have taken great care to strip out second-order effects like that.
guidelines should be clear and transparent and act across all companies. I accept a price hike if I've made a claim and I was at fault. I don't accept that someone else in the same position with a different company does not have to pay anything more than they did last year
Some countries previously had the sort of system you advocate. Almost universally they have moved to the current free market system because it's perceived as fairer, it takes out cost for consumers and encourages better driving.
If you, personally, had to cover the potential costs of repairing the damage caused by a random seventeen year old in a car how much would you want to charge? Bear in mind you could be liable for tens or hundreds of thousands of pounds, worst case.
Tens of millions. The most expensive claim I'm aware of in the UK was the Selby rail crash, which cost about £50m. A claim of £10m+ is easy to imagine - a 20year-old driving four mates who crashes into another car with four young people. Everyone left in electric wheelchairs in need of 24-hour carefor the rest of their natural lives.
My point was to the OP that it doesn't seem fair (to him) that a new driver has to pay £4k per year to insure a car but that's just how it is given the cost of claims made.
Yup. Insurance companies in aggregate make almost nothing from motor insurance - obviously some firms do or nobody would sell it. Distributors make a few tens of pounds per policy. The overwhelming bulk of premiums go in claims.