Don’t remember specifically trying it in a 32, but I have used ffff in handguns and some 45 caliber flintlock and caplock rifles. Nothing remarkable was noted. No buildings collapsed nor did the earth open up. For what is worth I know that when Lyman tested revolver loads for the data in their Black Powder Hanbook back in the 1970s they tested with both fff and ffff. Not for rifle data, but still used in pistols without worry or extra caution.
Pulled down an original Smith Carbine round years ago and remember the powder we found was finer than the ffff we had. Now the Smith an early attemp at a breech loading cartridge gun, so not sure of significance in powders used in muzzleloaders, back in the day or now. Just an observation.
When I was starting years ago, the ‘rule’ from gun makers was ffff for priming, fff for main charge in 45 caliber and under, ff above 45 caliber. Interesting that today I generally use fff for all my 32, 45, 50, 54, 58 and 62 caliber rifles plus 12 gauge shotguns. I may use different powder for a particular gun if there performance improvement, but typically not the case.