Yes but exercising your right if it makes it safer, by preventing bad overtakes, is worth it, even if some drivers are slowed by a few seconds.They may have a right, but that doesn't make it sensible or safe. That's the sort of approach that sees riders stretchered away, bones sticking out their skin, shouting "but it was MY right of way!"
Having a "right" means jack when it comes to life expectancy on the road.
My safety trumps your convenience.