So it's a steel expander cone wedged inside a steel steerer tube. A different problem.
If you're able to get the fork off the bike (and you can, because you've had the headset apart), a combination of gentle heat and percussion should shift it. Rest a bit of rod or tube against the expander inside the steerer and hit the other end of the rod with a hammer.
But as you say, until you want to adjust the stem height, it's not a problem.
(You know you're not going to sleep properly until it's done, though, don't you?)