You are, as you suspect, wrong.
My argument is quite simply that when drivers are driving in a shared space which is also open to pedestrians, cyclists, sheep, horses and random other legal road users, then because physics says they're the most intrinsically dangerous people, the moral obligation is on them to take extra care correspondingly. Not on the horses and pedestrians to get out of their way.
If someone walks through a shopping mall carrying a bale of barbed wire, they would take more care, yes? Because by carrying it they are increasing the danger to anyone who bumps into them. If you collided with someone carrying that bale who was not taking appropriate extra care, you would reasonably be upset. If the result was they you sustained an injury and they got away without, you might say that was unfair, wouldn't you? So, substitute the mall for a road and the bale for a car, how does that change the picture? It doesn't, unless deep down you believe that the roads are not really shared spaces