If you have ever crossed the Fens into a headwind would not ridicule the OP.
It is better to be more aerodynamic than less, but when touring fully-loaded, that isn't really an option.
For ultralight touring there is no need for racks and panniers, or even a multitude of frame bags. Get a large Carradice saddlebag, located behind your legs in air that is already disturbed. It has minimal aerodynamic drag and a good weight/volume ratio.
If you must have panniers, the Tubus Fly rack is thinner than most.
Bar bags are handy for valuables and maps, but don't have any aero features. You could probably make one with a curved front.
Aero-bars are surprisingly useful on tour. Many Euro MTB-tourers clip them to flats for a more aero position, but the base bars are not as low as in TT or tri bikes. The main advantage is getting the hands close to avoid a parachute effect. In windy parts of Holland, some riders fit aerobars to their Dutch utility/ hybrid bikes,
