So why do 'some' men only have little ones?
I reckon it's because pretty soon cave women realised that there are more important things about a man.
I don't know of any modern societies where a dominant male gets to do all the mating, and we don't know that happened in the past, so most men got the chance to mate and pass on their genes. As soon as we started to wear clothes habitually, and have social conventions about who could marry who, it all went out of the window anyway.
It could also be that a larger penis size was genetically linked to some other unrelated genetic advantage, and it wasn't actually selected for at all. (For example, lots of white cats are deaf, because the genes for white and deafness are linked. The whiteness doesn't cause the deafness, or vice versa, but if you select for whiteness, you could increase the incidence of deafness.)
I don't know that a preference in a modern sample is necessarily deeply hidden in the psyche. It's just possible that those women thought they ought to prefer a certain thing, and reacted accordingly...