Sounds like you are dealing with an off center QLA, or as @Grenadier1758 suggested, some sharp edges.
Or you go right into sharp edge/burr removal. To this I would suggest cleaning the bore with something like Scoth-Brite. It won’t hurt anything and stands a good chance of helping to remove any sharp edges or burrs where the QLA counterbore and the bore meet. Mike Bellevue (@duelist1954) has a good video out showing his version of the process. I have used a similar method for some time, but Mike explains his very well. I started doing it after speaking to Don Getz (Getz Barrel) years ago before he passed, and he recommended using the green (600 grit) Scotch-Brite for smoothing up barrels that were cutting patches. Said it wouldn’t hurt the bore.
Unless you have access to a lathe to bore out or cut off the QLA, I would suggest trying to clean up the sharp edges first. If you have a borescope, time to take a look. A way to check for patch damage at the muzzle is to drop a short piece of brass rod (maybe 6”-12” long, 3/8” diameter for example) down the bore BEFORE loading a patched round ball and pushing ball a few inches down the bore. Then by rotating the barrel from muzzle up to muzzle down the brass rod works like a slide hammer to push the patch ball out so you can examine the patch.The design left sharp lands at the end of the counterbore that tore up patches and in practice did not improve accuracy.
Or you go right into sharp edge/burr removal. To this I would suggest cleaning the bore with something like Scoth-Brite. It won’t hurt anything and stands a good chance of helping to remove any sharp edges or burrs where the QLA counterbore and the bore meet. Mike Bellevue (@duelist1954) has a good video out showing his version of the process. I have used a similar method for some time, but Mike explains his very well. I started doing it after speaking to Don Getz (Getz Barrel) years ago before he passed, and he recommended using the green (600 grit) Scotch-Brite for smoothing up barrels that were cutting patches. Said it wouldn’t hurt the bore.