I used a rear-viz wrist mirror. It's a pretty high end one nothing like the one above. Very adjustable.
On flat bars with bar ends it was pretty easy to place in position and of course, is infinitely adjustable by simply moving the wrist.
I never had a problem with it for gloves or any other clothing. In cold or wet weather it simply went on top.
My model was quite large (a few years ago now, I believe current models are a lot slimmer) and I found it uncomfortable on long days because it needed to be pretty tight. For commutes in busy traffic I'd consider it superior to a mirror attached to the handlebars because of its versatility - adjust the wrist and get a whole new view.
As regards an off, I'd have been pretty confident that it would either snap off or fold up on impact. I'd be more worried about the design of the one above.