I was having the exact same thoughts myself as the op. Using my cannondale hardtail for a little bit of off road touring necessitated a rack.
Did fret at first, then decided actually liked how it looked. F*ck what anyone else thinks as I cycle by, I got my bike to ride, not to be judged by fat gits smoking in corsas at traffic lights, just driven back from picking up their latest trendy mtb mag.
Stick ur rack on, ride it and enjoy it. If ur going to do some technical riding take it off for a bit, or bungee a stick onto it to poke the eye of the first reprobate to give you 'advice' about taking it off.
Sure it has effected the handling but I don't intend on tearing the ass out of it down mtb trails whilst loaded, unladen there is no difference to without a rack.
Plus I can pack the hexy stove and plenty of cake, make a fresh brew in the middle of a claggy fifty miler on salisbury plain, something to soften the heart of even the most ardent mtb minimalist.
Sound a bit militant there, sorry! Had no idea I harboured so much love for my rack...