you should get rid because they harm honey bees.
I don't think that's true at all. Wasps are important pollinators, especially in towns, and some plants are pollinated exclusively by wasps.
I get a wasp's nest most years. They never cause me any problem, so I leave them be.