what exactly do you mean by rough?
either there may be grit in there or else there races have become pitted - the latter is more notchy since as you turn the bars, the ball bearing will tend to get momerntariyl stuck in the pits in the race.
you need to take it apart for a looksee.
if races and bearing are fine, quick clean witrh white spirit/paraffin/petrol, regrease and reassamble.