One of the World's fastest cyclists Graeme O'bree used to time trial on a fixed wheel, no time wasted changing gear and lighter bike. Why then is everyone so concerned nowadays about how to get more gears?
Whilst I agree that selecting the right gear is essential to produce max power and gearshifts are now quicker with indexing, I think you are a bit slower if the gear is changed. For a start your aero isn't as good, heavier bike, more rotating weight, and power can't be applied at all times.
More gears are probably ok for touring though.