The cold will have an impact on your knees - I'm not quite sure the temperature range involved but you can damage cartilage / tendons in your knee from riding with them uncovered when its cold (they become less supple and elastic). The problem here is that temperature wise you could still feel alright and you might not notice the damage being done - it is accumulative and might take a few years to suffer the effects.
Seriously, can you cite any evidence that cartilage or tendon damage to a healthy knee can be caused by riding in the cold?
I'm not picking an argument, I'd genuinely like to know. I spent some time searching and found nothing. Just stuff I already knew about barometric pressure affecting pre existing conditions.
I concluded that it was just a meme, but my mind remains open to further evidence.
But as far as the OP goes ... try knee warmers. If nothing else, they will warm your knees.