Are you planning on leaving it bright, or are you going to brown/blue it? From what I've read, browning/bluing doesn't require too fine a finish.
As for finishing it bright, it may be that you progressed to the finer stages of sanding too soon. If you keep all your sanding in the same direction, as you progressively move to finer grit, you shouldn't see a swirl, as any swirls would have been removed by the heavier grit. You may need to try a polishing compound once you get to a fine enough grit. Just a guess, though. I'm no expert.