I've tried shooting undersized balls.
Most antique Northwest guns were nominally 24 gauge, which would be .579", but by actual measurement most come closer to .60 caliber. I believe a true 20 gauge would be .615". It is my understanding that the military standard for round balls in smoothbores in the 18th and 19th century was .050" under bore size. That is not a typo, but is based on focused research I have done. This seems egregiously undersized to us 21st century hobbyists, to the point that some modern shooters may refuse to believe it. However, that's what the references appear to indicate.
You also see references to a "half ounce" trade ball. A true half-ounce ball would run 32 to the pound, and would measure .526". However, moulds recovered from old trading post sites throw balls closer to .550", which would be 28 gauge (28 to the pound), and at least one traveler (the Earl of Southesk) in Rupert's Land stated that balls of 28 to the pound were available virtually everywhere. The Hudson's Bay Company (HBC) did carry a few other ball sizes, but based on the reading I've done, I am convinced that the "half ounce trade ball" was around .550". This corresponds neatly with the military standard, in that a .550" ball in a .60 caliber gun would be .050" under bore size.
In The Journal of a Trapper, Osborne Russell described a firefight with native people and mentioned pieces of blanket that they used as "wadding" raining down on his band of trappers. I have often wondered if the blanket material was actually used as patching rather than "wadding," but we will likely never know. A thick fabric would make a good patch for an undersized ball. However, I have seen references to native people using shredded bark and (in the south) Spanish moss for wadding, and an old trade musket recovered from the Suwannee River a few years ago was still loaded, with lab analysis showing the wadding was palmetto fiber.
In any event, I took my .24 gauge/.58 caliber Caywood Northwest gun to the range one day with a few .550" balls. These would be "only" .030" undersized. I brought along a supply of palmetto fiber that I had sourced out in the woods locally, as well as some shredded inner bark of juniper. I shot the .550" balls, and loaded powder, wad, ball, and smaller wad.
Accuracy was terrible, all over the target back at 25 yards. I admit to being a poor shot, but this was bad shooting, even for me. The worst part, though, was fouling buildup. The gun had to be wiped every five shots or so, or it became difficult to load, even with a bare ball. I am sure Indians did wipe their guns, as coil worms, or "wipers," were in demand from the traders. I know of at least one period painting by Carl Bodmer that shows a native hunter carrying a spare rod with a wiper affixed to it, and there is at least one old trade musket in the NMAI which is accompanied by a rod with a coil worm affixed to one end, and I believe it has a jag cut into the wood on the other end.
I would like to point out that both shredded juniper bark and palmetto fiber (and dead Spanish moss) all make excellent tinder for flint and steel fire starting. That is, they are quite flammable.
Anyway, I like to actually try things the way they were done in the past, and I would encourage anybody to do the research and try it themselves. I don't like to assume any more than is necessary. My conclusions from this experiment, shooting undersized balls with locally sourced, documented natural fiber wadding, are that accuracy is suboptimal and fouling buildup is excessive. I suspect the flammable nature of the wadding may have contributed to carbon buildup and fouling, but that is conjecture. I would like to try wool fabric as patching, as wool is less flammable, but I have not tried it yet. I would also speculate the there would have been a substantial loss of energy from "blow by" around the undersized ball, but I did not have a chronograph to prove it.
Again, I would encourage you to do some research and experiment... and report your results! However, based on my own experience, I would try to find a more accurate and cleaner-shooting load for competition or hunting. Ethics regarding hunting were different back then. Meat for the pot, or a hide, was what they wanted back then, however they could get it. A quick, humane kill is what we want in hunting now, and I would not recommend this type of load for that purpose. Several years ago, on another forum, a fellow reported going on a bison hunt with his trade gun, loading as they did back in the day with an undersized ball. He used dried grass for wadding. If I remember correctly, he hit the animal six times, and the "hunt" consumed about eight hours from the first hit until the bison finally went down. Being period correct is one thing, but I think I would have opted for a gun and a load that would provide a cleaner kill.
Best regards,
Notchy Bob