I'm sure that some cyclists don't like the notion of an unrestricted e-bike being faster than them with less effort. And it's weird to be struggling up a hill, while seeing someone on an e-bike effortlessly (comparatively) going past you.
But that isn't my gripe, not I suspect is it that of many others. While this forum may be full of people who can routinely exceed 25kph average speeds, that just isn't true of the general cycling population. I am very happy to get to that average speed on a longer run and while some people blast past me, mostly (in urban areas) I pass the many leisure and occasional cyclists who make up the majority round here.
Bikes are tolerated in part because they are generally relatively slow, and don't do much damage. A landscape suddenly cluttered with things that look like bikes but travel far faster is a recipe for disaster. The voices calling for mandatory testing and registration would be amplified every time one of these wiped out some child or older person. Change the perception of bikes to speed machine and all of us will suffer. And don't kid yourself that these ebikes will only be acquired by the responsible.
Sure you can go along at breakneck speed in your regular bike, but most can't. Change that to 'most can' and accident rates will soar. Cycling is currently a largely unregulated pleasure - derestricted ebikes will eventually kill that.