They are so comfy, they dont ride up or pull down so, for me, compared to my previous jogging bottoms I can get on with cycling rather than adjusting my trousers which must be a safety benefit.
The main thing is aerodynamics, if you have pants and coat flapping in the breeze it makes a hell of difference, especially in a headwind situation, might as well make the most of the limited aerodynamics we have rather than compounding the problem, it does depend on speed as the faster you go the more noticeable the aerodynamic effect, so to squeeze a bit more speed out they are worthwhile, its part of the reason Roadies overtake commuters and we wonder how seeing as you only see most of them at the weekend.
With regards going to the loo, bit more complicated but not much of an imposition.
Its just the wearing of them invites comments sometimes, you feel a bit self concious but the benefits make it worthwhile.