If you're talking about brown oily stuff on the rag, it almost sounds normal. It depends on a few things, like how often you've wiped it down since browning, how heavy is the rust residue on the rag, and if it was a really heavy brown job. You may still be wiping off the loose coating.
If it still looks good after you wipe it down, I wouldn't worry about it. Sooner or later you're going to wipe off the "fuzz". If it gets to the point where you see metal coming through, and don't like it (some folks like the worn look), then it's time to consider re-browning the gun.
I'm pretty sure the acid effect was killed after you soaked the barrel in oil for a day and the fact that you're kept it oiled for a year. If it's a pinned longrifle, then you might consider removing the barrel to lube the hidden part, at least until the residue ceases.
Here's something I do after browning a gun. I get a pinch of 0000 steel wool in between my thumb and forefinger, oil the barrel a bunch, then gently scrub the flats, being careful to stay away from the corners. You might try that method to loosen the fuzz. Start on the hidden part of the barrel to get a feel for it.
I don't ever neutralize the browning, but then again I live in the desert where humidity is not a problem. Good luck. Bill
