Not really the same, but I took 32c tyres off my Surly in favour of 42c tyres, and there was no difference in average speed on my commute - the ride was a lot comfier though
You might get some benefit from having a slick tyre rather than a treaded tyre (better grip on road, quieter ride).
If it were my choice, I'd stick with the bigger tyre size, but choose a slick (Continental Comfort Contact, maybe?) If it's the sort of thing that will niggle away at you until you've tried it, go for it though - tuning your bike to you, and finding what works and doesn't for your ride is all part of the fun, after all
.