I stopped wearing one several years ago, but still donate.
My logic is that if they cost say 3p to produce, and you donate £1, all of a sudden, your donation is worth 97p. As they say at Tesco, every little helps. In fairness, where I used to "buy" a poppy once a year and feel happy that I had "done my bit", I probably now actually put more in the tubs in total.
I dont feel the NEED to wear it to tell everyone else I meet in life that I have donated, any more than I advertise that I have given to other charities.
I should also point out that I dont have issues with the people who do take one and wear it. Each to their own and all that.