Making a sweeping generalisation (I've only tried this for 1 day & forgive me if you don't come in under this category) but, it would seem that the people who insist it's dangerous are the ones who haven't anything to back up their argument.
I agree. Although fortunately we haven't had the kind of comments I've seen on previous threads where after I've eruditely explained why I feel riding with headphones is safe, you get one-liner comments like "People who wear headphones whilst cycling are bloody idiots and they are going to get themselves killed", which I find so frustrating.