I teach at uni, and though I have some old panniers in a cupboard somewhere, I usually use a backpack in any case. My commute is short (just over 3 miles), and I seldom have more than a few kilos of paperwork (marking etc).* If I had any more, or a longer trip, I'd def use the panniers more as I'm one of the sweatier members of this parish and get "backpack back" pretty easily!
A rack and panniers weigh very little - probably the same as an empty backpack at a guess - so weight shouldn't be an issue. It's the hassle, for me, as my old pannier straps take about 3-4 mins to attach or detach. Modern panniers though prob clip off simply & quickly, so again prob not that much hassle. Is there something else you're avoiding? Destroying the clean lines of your road bike, etc etc?
__________________
* In fact, that's just changed again as our uni has moved to online marking of almost all coursework... which means no more cycling down to the park to do some marking on the grass on sunny days... boo.