You should be able to pull the leg warmers all the way up until they rub against the pad of your shorts without issues.
That'd be too high for me. Ime, and seemingly that of the OP too, that'd cause irritation.
I go for a slightly lower position; high enough to stay put yet low enough to not irritate. I also make sure there's not any gather behind the knees and that zips are not sitting on my ankles. I try to distribute any 'slack' evenly but it usually finds its own home anyway.
Sadly, I found it uncomfortable to allow too much gather at the ankle and I need to distribute it a little. But then I find that if the tongues on my shoes aren't 'just so' then they can irritate as well! Guess I'm just a picky bugger!
Don't have any problems around the shoe/ankle area so will leave the excess material there where it does no harm.
shorts too at least hehe sorry