While a rack and panniers aren't usually necessary on an "audax" event, personally, I wouldn't want to try an early-season brevet with just a pair of arm-warmers stuffed in my jersey pocket.
In fact, on PBP '07 (August) it was the riders who tried that approach that were most often seen being carted away on stretchers, bundled into ambulances and taken away to be treated for hypothermia. One chap I saw was so far gone, he was completely unable to respond to shouted commands, even though he was still sitting upright and had his eyes open!
If you have access to the '07 DVD, check out the shivering zombies seen at controls (you might try YouTube for some clips).
So, my preference is for some kind of small racktop or handlebar bag large enough to carry clothing and kit to get me through cold nights and inclement weather ... and yes, canvas and leather is very nice if you can afford it.
There seems to be some confusion as to what constitutes an "audax" ride, perhaps due to the misnomer in the UK. An audax ride is run under different rules than a randonnee, as has been noted in another thread.
Official Audax Club Parisien brevet distances begin at 200km (125 mi) and go through 300, 400, 600 km, etc. PBP is 1200km. 100 km is not a recognized brevet. Shorter introductory rides are called "populaires" here in Canada and do not count toward any awards, standings or medals, other than local club pins we give out to encourage participation.
I'll start a 200k on a nice day with little chance of a change in weather carrying minimum tools and kit, but anything over that requires a "be prepared" approach.
Most experienced marathon riders usually end up with a bike that is comfortable over the long haul.
There is a fuller explanation and discussion of gear beginning at VeloWeb's
Rando hub