You obviously have a council that thinks that every driver that they give a taxi licence to can SAFELY transport a client from A to B. My own evidence of the standard of most drivers does not back this up. Not all councils require this test, but more and more do. Those that do join scheme, then make it a requirement of licence renewal.
Just because someone can pass a CRB check, does not mean they are a 'fit and proper person'. It just means they haven't been caught. Or, if they have, it happened a specific time in the past, so that it is now not considered a problem to the public.
I do agree that one persons uncorroborated complaint is not proof. As I said, several 'serious complaints', would possibly be needed.
As you hint that you are a (hobby!) taxi driver, would you not like to have your driving assessed by a qualified examiner, to make sure that your driving is 'up to scratch', so to speak! Showing a little professional pride.
Maybe you'd like to visit this
www.gov.uk/dsa-taxi-driving-test and see what's required.