Nothing wrong with 23mm. Why run tractor tyres ? 23mm roll the best in all conditions.
Tyre size, the facts:
a) At the same pressure, a fatter tyre of the same type rolls better than a narrow one, so GP4S 28 @ 100psi is faster than a GP4s 23 @ 100psi
b) The same tyre will roll better at a higher pressure, so GP4S 25 @ 120psi is faster than GP4S 25 @ 80 psi
c) A narrow tyre is lighter and more aerodynamic than a fat one
d) On a rough road (aren't they all), bump losses are higher at high pressure.
e) A heavy, heavily treaded tyre rolls much worse than a light, lightly treaded tyre.
f) Maximum tyre pressure is lower the fatter the tyre
Bump losses are the energy required to jiggle all your muscle, flab etc about. This can absorb many times more energy than is lost to rolling resistance - consider how quickly you lose speed if you freewheel onto cobbles or another rough surface. You can more or less equate bump losses to comfort.
So, (a) and (d) favour a fatter tyre, (b) and (c) favour a narrow one. (e) just means don't use a mountain bike tyre on the road unless you fancy a workout.
How smooth are the roads you ride on? I use 28s at around 85psi.