I do likewise and occasionally top it up from the shaver adaptor points in the camp toilet block. I choose a quiet camp and the phone isn't state of the art. I do the same with camera batteries if necessary. Most campsites that I use are small and full of similar kinds of people. There is a large element of trust from mobile home tourers who leave tables, chairs, dryers, bikes unattended, to bike tourers who leave cooking, sleeping stuff, clothes in an unlocked tent.
Within reason, the whole thing works on trust and I've never had anything nicked from a camp.