sub-700x23c tyres are the fast rolling tyres but they tend to be less puncture proof than their wider siblings. Now interestingly one of the properties required to make a tyre roll fast is suppleness & as the tyre its self makes a huge impact on the ride quality, could mean that going to a 20c tyre makes for a more comfortable ride.
As a general rule of thumb if a 23c tyre bulges out a little over the edge of the rim a 20c tyre will go on just fine, you can always measure the inside of the rim(the distance marked 18
here), if it's more than 15mm then you're really pushing the limits of the tyre. The normal range for a inner tube is 19-25c but there are usually markings on the tube to tell you what size it is.