Perfectly legal, just as it is to push your bike along a pavement. The only thing that could have made a difference would have been if it was a permissive path (orange dotted line on 1:25k maps) where the landowner has given permission so could theoretically also withdraw it.
Farmer just being stupid imho