That's your head's problem. Accept it and move on.
I have two smoothbore flintlocks, both "20 gauge" both shoot better with a ball sandwiched between wads than they do with patched ball.
The 1st one I got shot well with a patched ball, and I was too new to question anything, the previous owner had given me his load data.
The second one would not shoot a good group with a tightly patched ball. Finally, having read so much about folks shooting well with no patch and just wads,,, I tried a looser patch/ball combo,,,, accuracy improved..... hmmmm
Finally based on that, I tried; powder, thin card, lubed felt wad, ball, two thin cards to hold it all in.... even better. Played with powder charges. Final outcome was sub 4 inch groups at 50 yards. Sometimes sub 3 inch off the bench.
Point is,
smoothbores are not rifles. One needs to get out of ones head, through the rifle "rules" away, embrace the black magic that os smoothbore shooting, and shoot what works.... even if it defies "logic" and common "wisdom."