I know you don't wear a helmet, and I know the reasons why you don't, but have you thought about a bump cap?
That's not an invitation for a "helmet thread", we all know the pros and cons of a cycling lid, but bump caps are the same price as a cycling cap with a solid liner.