There's a fairly respectable climb out of Port Eynon in the first place. But you're right about the climbs up from Oxwich - there are a couple of horrors. If you do find yourself down at Oxwich (and the marsh there is lovely to cycle along on a misty morning or evening) the least painful way up is
this one, which brings you almost back home.
My first choice from there for a sub-20-mile on-road loop would be
this, anticlockwise. Other than Port Eynon, there are two significant climbs - a bit of a drag through Reynoldston up over the ridge called Cefn Bryn, and a sharp uphill South from the Britannia Inn (worth a stop, especially on a sunny evening) at Llanmadoc. Both are rewarded with great descents, the stupendous hurtle down from Cefn Bryn to Cilibion being the perfect stretch to go for your downhill record. Once you're over the climb out of Llanmadoc it's downhill or flat as a rug all the way home.
This is your flat exit to North Gower, and a perfect cycling road it is too.
This (signposted Gower Heritage Centre from the S Gower Road) is another useful, flat connection between S & N, if you don't mind rustic sufaces and having to faff about with a couple of gates. The couple of fields at the northern end of it are not a right of way for cyclists despite being perfectly suitable, but you can get off and walk if this bothers you.
The Cilonnen Road is one not to be missed, and although
really it's best ridden E to W, I'll advise you otherwise from your starting point. It's a wide road along a ridge with fantastic views to both sides, very little traffic, and no terribly challenging climbs. You could then treat yourself to
this wonderful, fast, swoopy descent, and link it up via the B4295 with the beautiful (and flat)
Marsh Road from Crofty back to Llanrhidian (note: it floods on very high tides). There's a short but rather sharp climb out of Llanrhidian, which you could always put off for a bit by stopping for a pint at the Dolphin.
This is probably your best route back from Llanrhidian.
I've not really mentioned the rest of South Gower much because it's quite inaccessible without spending a lot of time on the South Gower road - this is quite nice if you're the more assertive sort of road cyclist, but lots of people don't like it at all. As for other connecting roads,
this road should be ridden strictly N to S unless you're a masochist, but don't do
this one from N to S if you can help it.
This (in either direction) is one of my favourite roads when in the mood to add on a couple of gratuitous scenic miles.
Rhossili and Llangennith are worth a visit, but are both dead-ends unless you go off-road. And I know we're still talking about on-road routes, but as you have an mtb it's a shame not to go along Cefn Bryn ridge for the views. From the car park at the top of the ridge just N of the King Arthur Hotel in Reynoldston there's a bridleway that climbs relatively gently heading E along the ridge. Another thing to note when further afield is that there's a very useful flattish cycle path (former railway) that runs from Blackpill on Swansea Bay all the way to Gowerton, and you can pick it up at the Railway Inn in Killay or at Dunvant - it's the minimum-effort way of getting to any of the three main Gower roads from the Bay.