The rough spot occurs because the petroleum oil burns down to tars, and sticks to the barrel. Don't use oil for cleaning. You can use it for storage, but I don't even use it for that anymore. I use Wonderlube, altho I have ballistoil to try. Either way, you have to use alcohol to clean the oil out completely before loading and firing. Then use a vegetable based lube or spit on your patches. Olive oil works very well. So does Wonderlube, ( Bore Butter, NL 1000- they are all the same product.) If you are worried about contaminating the powder charge, then use an OP wad, a filler( wasp or hornet's nest) another patch folded and run down on the powder, Corn meal, or some other fine grain, synthetic fillers like Puff-Lon, etc. to protect the powder from contamination. Skimping on the liquid patch lube is why the crud is not being dissolved and taken out.
HINT: There are no short cuts. Take your time and do it right. If there was a faster way to reload or clean a barrel, we would have heard about it more than 100 years ago, when No one shot smokeless powder: it hadn't been invented, yet!