I'm afraid pedestrians have every right to walk on the cycle part of a shared path, and I suspect, even on a dedicated cycle path. Yes, it's annoying when they wander across, but it is us cyclists who must stay on our bit. If you're not prepared to cycle along in a way that allows you to give way safely to pedestrians (IE slowly and covering the brakes), use the road. Peds who don't also cycle are not used to having to think about where they walk, unless it's to cross the road, and I'm sure very often they have no idea they are on a cycle path, even if it's well signed. Whenever my Mum comes to visit and we walk along the river, I have to remind her to keep to one side - she's not stupid, or willfully ignorant, she's just not used to shared paths.
That said, I've said things before now, usually along the lines of "watch where you're going!" as they step out in front of me...