Panniers are the way to go for work luggage. One should carry all you need, two will provide some shopping room on the way back. The best way to attach panniers to rack is with a quick-release locking system like Ortleib or Rixen and Kaul. Hook and elastic is a bit of a faff every time, and some of the hook and lock systems drop off when the mechanism fails.
Carry your shackle lock on the rack top, held in place by a bunjie chord.
You may need to carry a repair kit, esp puncture repair, which usually means a spare inner tube. For 4 miles, you are just over my limit for kit-free riding, but not long enough to need the ultimate in speed and efficiency, so ideal candidate for Schwalbe Marathon Plus tyres, the toughest inflatable ones on the market.
The other accessories most commuters need are lights, gloves, [helmet], mudguards, trouser clips, bright windproof, bright waterproof top, waterproof trousers, neck loop/buff.
You should be able to ride in work clothes. You either ride fast, sweat, cool down, change, or you ride slow and go straight to your desk. It takes about the same time both ways. If anything the latter strategy is quicker.