Update on this.
Got fed up with this and put the bike to one side to think about whether I was going to keep it.
In the meantime I bought a Diverge Comp frame and wheels from
ebay to build up, after I finished I fitted 35mm Voyager Hyper tyres, went for shakedown ride and all was well.
I then decided to fit the 28mm Clement LCV tan wall tyres from the PX because they complimented the colour scheme, straight away the handling changed and was more twitchy, not as bad as the PX but noticeable. So I then fitted 28mm Ultremo ZX tyres back on the PX and immediately I could tell that things were back as they should be regarding the handling, which was much improved.
I still don't know how this is happening as the ZX tyres are just as light as the LCV's. I also tried different pressures over time to rule that out as well so I am a bit puzzled