YES, I have used shot loads in a smoothbore pistol. I had it loaded that way to take care of snakes where I hiked.
NO, at least from my experiences, the load was effective on snakes at a couple yards away, but I wouldn't depend on it for hunting game unless it was within 10yds--maybe closer. I don't think using it would be fair (sporting) to the game or to yourself.
NOTE: Some smoothbore pistols are only smooth by virtue of not being rifled... but they still have tooling marks inside (may looks like they have rings the length of the barrel). In an exceptionally smooth barrel, you will get somewhat better performance than I did. (My gun did NOT have a polished & smooth barrel, as some guns that I've ever had.)
You would also get a better shot pattern, IF you have a caliber/gauge that you can use the proper sized wads that are typically used in muzzleloading shotguns. The more care you take, the better the shot pattern "could" be; BUT I still wouldn't consider it "sporting" for game hunting.
Are you familiar with loading shot loads for muzzleloading long guns?
It sounds as though you haven't as yet purchased your smoothbore flintlock pistol. Are you basing your decision on it's ability to handle a shot load, or something other?
Shoot Safely!
WV_Hillbilly