I'm going to regret this but...
With reduced reflectivity and contrast (i.e. dark clothes) you are less likely to spot a pedestrian or cyclist - that's just basic physics. There will always be circumstances where lights and high vis make very little difference. Most of us don't stay in one spot though. If I am in central London, my lights and high-vis equipment don't really add to my visibiility. The place is well lit. However by having elements of differing contrast (my coat and shorts are black and my bike isn't particularly colourful) I still thnk that I am noticeable enough. The only place I am truly invisible is standing on the concourse at Waterloo Station.
Getting back to Surrey however suddenly the roads are a lot darker. Many of the residential roads have low throw streetlights and some quite long dark spots. This is where the reflective material of my high vis rucksack, gloves and white helmet are useful, along with the bike lights.
Is there any evidence that wearing a reflective jacket improves the safety of the cyclist? Well yes, there does seem to be:
https://www.sciencedirect.com/science/article/pii/S0925753517313528
Of course there have been numerous studies suggesting the opposite, however they were not randomised controlled trials, but instead based on analysis of accident data (the linked study does go into this).
So, on balance, it seems to me that it is better to encourage people to wear clothing that at best may reduce the incidence of having an accident rather than to dismiss it out of hand. Being possibly better off is to my mind better than not having that possibility.
This argument goes around and around. Should we use lights? Should we wear hi-vis? Should we wear helmets?. None of these things have actively and definitively been shown in a scientifically controlled randomised trial to make life more dangerous for the cyclist. If you don't want to wear them - that's fine. I think there is sufficient evidence to show that on balance it's probably better to make some sort of effort.