Hello from Texas!
I too have always had the same results in getting a little brown on the patch after cleaning. Never was too concerned about it either and I suppose if you coat the inside with a good lube after cleaning it will still be ok to put the gun away for a while.
I did start using some stuff called Black Solve I think it is. Believe you can get it from Dixie. I was given some by ol' John L. Hinnant one time to try. Actually John cleaned my rifle for me with the stuff. It works incredibly well and after about 10 minutes of letting it sit and running a patch up and down for a bit the patch will come out clean! And I mean CLEAN with none of the brown residue on a white patch. The stuff is amazing. I was hesitant in adapting to it because I thought it was some new fangled cleaner John had found but he told me that Dixie had been selling the stuff since the 1960's. Well that was good enough for me and the last time I bought some I bought enough to last a while, about six bottles. I use the stuff for a patch lube also in shooting in the area competitions and it is amazing.
Anyway for what it is worth that is what I started using to get my barrel whistle clean without all the scrubbing, hot soapy water, and all the other chemicals that I used to use.
So to answer your question of how clean a patch should be, it should be as clean as you can get it I suppose. Mine is whistle clean now with very little effort at all.
rabbit03