Whilst that may well have been the cause for you on that occasion Carl, it's not the "only" reason this happens. Happened to me at the start of a race the other day, but I'd been riding in a group ride in Zwift for a good half hour immediately before and everything was fine. I didn't change the difficulty setting or anything in between exiting the group ride and joining the race. It was quickly solved for me by a quick switch off and on again of the plug for the trainer though.
I have experienced similar from time to time in Bsim, but very weirdly it would sometimes happen every time I rode a particular route - in fact I remember one route on a chaingang league some time ago where several of us went up several fitness levels as we all got a boost on that route. Very odd!