Just add my four-penneth. You are allowed to follow your own route but as suggested, you must pass controls or information controls. However, information controls are now discouraged by AUK for new audax events due to their unpopularity amongst riders (a route proposal of mine was recently rejected for that reason). Secret controls are not particularly common - in fact the only time I have seen their use has been on sportives. The use of a secret control has to be notified to riders in advance of starting the ride.
Some riders have good reason to take an alternative route, for example, they may prefer to make a detour rather than include a large climb. However, route cards are usually constructed using the organiser's expertise of the picturesque or a need to channel riders away from dangerous and busy main road alternatives.
Some round the year audax rides, known as permanents, only list the control points with an expectation of riders designing their own routes - though many permanent route organisers do provide a route card or indicative suggestion. 'DIY audaxes' are as they say on the tin, Do It Yourself, where you develop your own personal route, building in controls but the distance has to be the shortest between the control nodes. Riding permanents or DIYs can be a lonely business unless you have a companion.
Going back to your point about going off road - I know of a club that focusses upon audax rides - they regularly go-off piste. In fact, I know of one audax where the route is partially off-road.