Could you describe more what you mean by "it feels like it's pulling"? If you're confident that the headset tension is set correctly then I'd start looking at tension of handle bars, is there any play in the front wheel (grab the rim and try moving it from side to side, check for play) etc.
The fact you said "there isn't any chance of doing no-hands" - That's often a sign that the compression is actually too tight, as it prevents the fork from tracking properly by itself..
If it's none of these things then I'd recommend taking it to a bike shop, as without actually looking at it in person I'm all out of ideas!