Perfectly legal, just as it is to push your bike along a pavement. The only thing that could have made a difference would have been if it was a permissive path (orange dotted line on 1:25k maps) where the landowner has given permission so could theoretically also withdraw it.
Farmer Giles was trying it on. You are allowed to do it, a lot of them don't like it and will put up unofficial signs that have no legal status or do tricks like padlocking gates and things illegally to discourage bikes/people. If he was being threatening I suggest getting in touch with the people who are in charge of the definitive map in that area and report him.