Golly gosh. It's hard to comment without seeing the thing.
But...
It depends on many factors.
I have frames from which the rear wheel tumbles as soon as I flick the QR and frams from which I have to wiggle and tease the wheel.
Having a think about it, I think the wiggle/tease frames are all (or have all been) steel. I am not a metallurgist, so this may just be hokum.
I'm guessing (by the topic heading) that this is a fixie question. The wheel in my fixie is certainly less inclined to pop out when undone, but mine's a road frame that I had cold pressed down to 120mm. It is snug. Also, steel seems to be a nicely 'alive' material.
I think it shouldn't be an issue as long as the wheel spins nicely when it is in position.
I'm slightly worried by the thought of a loose wheel being 'almost impossible' to shift. Is it just that it takes more heft than you'd expect?
I realise I've rambled here, but it was meant to help. If it works OK when assembled, I'd be inclined to live with it.