Ignore the neighbours. Most of us are too fat, to the extent that we now view overweight as normal.
I'm overweight but get comments about "all the exercise you do makes you look skinny". Including from my mum! I'd like to loose 2 stone, then what would the comments be?
If you feel comfortable, can do anything physical you want to and feel well what business is it of theirs? If you aren't too interested in neighbourly relations you could try telling them they're too fat and should take up cycling, swimming or running!