I'm fitter now than I ever thought I'd be when I first started, but the funny thing is - the fitter I get, the fitter the people I seem to meet!
Speed - I couldn't cope with the pace of even a 4th cat race.
As for distance - when I started doing 200 km audax rides, that was a big thing for me, but I met people who saw them as merely training for 300s, 400s, 600s, or even the 1200 km Paris-Brest-Paris. One guy told me that he didn't bother doing many 200s any more because he no longer found them challenging...
I came to the conclusion that ultra-distance** wasn't for me. I get too many aches and pains and I don't enjoy riding in the dark. Also - I want to have a life outside cycling. I have decided to do more sportives and try to improve my speed, and especially my climbing.
It is important to not be discouraged by those fitter than yourself. I did
The Pendle Pedal last year and came 191st out of 196 finishers but I'm not ashamed of that and just look at it as a target to beat. I'd like to be in the top half of the field this year, and maybe go for a decent time next year.
Andy - set yourself challenging but achievable targets. Work hard to get to that level, and then raise your sights for the next target. I'm organising a 50 mile forum ride in the Forest of Bowland for one of the last two weekends in June. It will be hilly, but the scenery is great. We will ride at the pace of the slowest rider, which is often mine! If you fancy coming along, how about making that your target for early summer?
** My definition of ultra-distance is anything much beyond about 200 miles or maybe 300 km. I know that I could cope with 300 km and it is doable in daylight hours in the summer.