I have just lightly read over the multiplicity of answers to your question. I am sure that my suggestions will, for the most part, just be a rehash of other's answers . First, change to 3f powder and reduce your charge by about 10% to start. 3f is a cleaner burning powder than 2f. Next, if I might borrow from another author "To swab or not to swab. Aye, that is the question. whether 'tis nobler in the minds of men to forgo the swab and use a mallet to seat thy ball on the charge or to take up a swab between shots to keep thy bore relatively clean and consistent from shot to shot and thus endeth the problem."
To borrow further from that same author "Is this a range rod which I see before me, the handle toward my hand? Come, let me clutch thee. I have thee not, and yet I see thee still. Art thou not, fatal vision, sensible to feeling as to sight? or art thou but a range rod of the mind, a false creation, proceeding from the heat-oppressed brain?"
Don't let that range rod be just a vision. Take hold of it and use it between shots to keep the crud from building up in your bore. As you have seen, this crud has a notable effect on your accuracy. Swab it out and be done with it.