I have yet to see a high use gun that did not have rust on the underside, if it was not periodically removed from the stock. That does not speak to the grit that accumulates on the wood.
Whether this junk is harmful to the barrel and stock is debatable. So also, which is the most damaging; leaving the gunk under the barrel or removing the barrel pins.
I am from the school that removes the barrel almost every time, drops it in a bucket of hot soapy water, pumps the water to remove the fowling, pours in clear hot water, runs a drying patches, WD -40 patches and finally a Ballistol patch.
This is quicker for me than trying to swab with numerous patches while keeping the water from running down between the barrel and wood. It also allows me to clean up the area around the touch hole nicely.
I have the advantage of having built these guns so assembling and disassembling them is not an issue.
In the field I will remove the lock and pour hot water down the barrel until it runs clear from the touch hole, follow with a drying patch, then WD -40, then a Balistol patch. This work until I can get home for a thorough cleaning.
The certainly is no one way to clean a muzzle loader.