It's a tricky one. However conscientious with observation a driver is, it's easy to miss a cyclist. It's often down to unfortunate timing. When driving, I've never knocked anyone off or for that matter, had a cyclist shout at me. As a keen commuter and recreational cyclist myself I feel my awareness is high. However, there have been a couple of moments where I've almost pulled out of a junction into a cyclist's path… almost. Sometimes a bike can just 'pop out' on you. Being for the most part silent (assuming daylight) the only real cues to their presence are visual.
Saying this, if cyclists are in a driver's considerations they're more likely to spot them. Blatantly obvious (apologies), but if you're not looking for something it's far less likely you'll see it. Even when something is directly in front of you, and very apparent were you to look back with an alternate mindset- if the brain's looking for something else… nope.
Maybe the answer therefore is more awareness on the driver's part; accepting cyclists are legitimate road users, appreciating their vulnerability and ensuring they consciously look for them.
On the cyclist's part- they have the advantage when it comes to observation: greater field of view, no musical distractions (normally), no glass between them and the sounds of the road, the exposure even provides a better overall 'feel' of traffic. That's why I believe there's perhaps more onus on the cyclist to observe traffic/peds and anticipate potentially dangerous situations. Good to assume that occasionally a car won't spot you and cycle in a manner appropriate for that assumption. It's all common sense stuff. With experience I find you know when someone's (cyclist, ped or driver) going to do something silly from all the little cues, and a situation is usually avoidable.
Sorry for rattling on, just awaiting a brief :-)
