Cycling in Edinburgh can be an awfully stressful experience for a beginner. The state of the roads is horrendous and the amount of peanuts on the roads is ridiculous, especially cab drivers.
I start enjoying my rides only after I'm past the bypass. Plenty of incredible routes with my favourite area being East Lothian.
About cycling clubs, I only know 2: "Ronde cc" that does Saturday rides starting from I think Stockbridge, and "Musselburgh rcc" which does club rides but on a Sunday. Don't have first hand experience about either of them but I'm sure they're friendly people and you won't be left behind.