To Victoria there are about three trains an hour, although one of those is quite close to an other, and takes longer (it stops a lot more). Realistically it's effectively two an hour, more trains if you don't mind ending up at London Bridge instead of Victoria.
It's only an hours journey, so unless you travels at the London end are a long way, or you take a very long time over your breakfast, or we have an awful lot of problems getting to Brighton, it would be hard not to be back by 1pm.
(Just had a quick look, and if you trying to get to Chelmsford, you could get there one minute after 1pm if you left at 10-49. This would be a pretty relaxed time to leave Brighton at).