I would say that there are a few issues here.
Firstly, I'd suggest rethinking this whole issue. The riders who are going past you are doing the same thing as you, enjoying their ride. They are not intentionally seeking targets to intimidate. If you can start to think of them as people, just like you, it may remove some of the feelings that you have towards them.
Secondly, whilst I agree with rear observation, it sounds as if you are enjoying a generally more relaxing pace (I do too) so would actually recommend a mirror as the need to constantly look over your shoulder is not exactly relaxing. I don't usually agree with rhwm.as on busy roads nothing beats direct eyeballing (in my view) but on segregated track like yours, perhaps it's the best solution.
Lastly, I would encourage you to work on your stabilisation if you feel you do weave about a bit, then that's hard work and requires a lot of concentration, grip, movement and generally lack of fun. Not to mention the impact it can have on other riders who wish to pass to continue their enjoyment.
Most of all, keep riding it's clearly a good thing and would be a shame for you to miss the activity because of feelings of tribalism or lack of confidence.