In anticipation of eventually getting round to some wheel building, or at least re-rimming and issue solving, am reading Roger Musson's book at the mo. He has some very straightforward things to say about whether lighter wheels will make you go faster.
And offers the alarming news (for some) that deep section carbon rims only have a chance of coming into play if you are going over 30mph.
Far more important, he sensibly reckons, is to maintain your bike and wheels well, particularly the wheel bearings.
Pretty sure you are right with your last two sentences in particular.