I don't do secret annoyance, why let something nibble away at you when you can share the pain.
I used to live in a terraced house with a back alley that intersected with another row of terraces. My bin had 8K on it the other street No 8 had just 8. More often than not they were home before me and took (I guess) the nearest 8 bin to them, leaving me with whichever one was left, no biggie for me. One day I was off work on bin day & took their bin in early on as it was nearest to me. I heard a rumbling later and found my bin by my back gate with a note taped to it saying 'Sick of you taking my bin, use your own you spekky tw*t'. Meh, sad git.
His GF collared me a few days later apologising profusely as she's been the one taking my bin in each week having no idea it was winding him up so much until he exploded that day. She said she'd told him to come round but he was too embarrassed to speak to me.