^^^ This is the important bit. Keep the enjoyable rides coming.
On my first "big" ride my previous best had been about 18 miles and I set myself the goal of riding the distance of a marathon. It was hard going and I was sore sitting down afterwards

but I felt so pleased with myself. Today I have four years of Imperial Century Challenges under my belt.
Build up to it and you'll be amazed at what you can achieve.
Yep, similarly, I set a marathon as my first "long distance" in 2014 - it took me over 2 hours, too...
I only surpassed it in 2018, when I started cycling "properly" again at uni. Went from 20 miles to 40 miles (up the A44 to Chipping Norton and back).
Then it was 50, which was the same Chippy run, plus spanning Oxford from ring road to ring road.
Then 60, stopping for lunch at Nando's Swindon.
And that increase was in the space of about 1 month...
Getting to a century took about 6 months, although no doubt I could have done it earlier. I actually only ever have done 6 centuries, 2 being >300km, 2 being >200km.