I've only ever had one "you could have signalled" issue where I felt unsafe to signal. The driver gave me some verbal abuse, but it I felt I had more reason to feel aggrieved than he.
It was a park car park, it was pretty predictable I "could" be turning in as it's a cycling route etc, well known amongst the dog walkers who drive there. The car park has two "accesses" and I've always assumed that they should work much the same way junctions do. One for in, one for out. When I drive there its what I do, because it's obvious it's set up that way to be honest, unfortunately no signs stating that.
This driver was coming out of the "access" that i would consider entrance only. The view of the road is heavily obstructed so I made no signal because it seems pointless and concentrated on being defensive instead, hands on brakes prepared not to be seen etc. Each other was seen so I proceeded to keep hands on the bars and turn in as it's over a dropped kerb, drain is subsiding, a bit pot holed etc I didn't give any signal at that point either, and as passing the driver decided to give some mouth. I don't see why as I didn't turn sharply from the gutter, position was always indicating I'm this far over on the road "for a reason" etc.
The only reason for any verbal is that driver couldn't inch out as he could if I was going straight out, like typical drivers who can't wait until you pass. Or I suppose if he knew I was turning in, he would have shot out in front of me which is another issue with signalling.
If I'd been in the drivers position, I wouldn't have felt any reason to feel aggrieved, so I never saw what the fuss was.
I don't often feel like talking to drivers, but this was one of those occasions where i wouldn't have minded, but I didn't bother.
As for when I'm traveling at traffic speed, I feel like I actually have more time and space because I'm not constantly looking over my shoulder for cars to 'near instantly' over take, and my area becomes more 'mine'.