OK, this response will perhaps sound like it's going off-topic, but believe me, it's dead-on topic, just using different equipment.
My home defence gun is a Mossberg Model 500 12 guage shotgun. The barrel installed in it is their fully rifled slug barrel. The load I use for practice is cheap old 2 3/4" #7 1/2's, while my more serious load is Winchester frangible double ought buck.
Why use this for home defence? Because at 7 yards, the pattern is MAN-SIZE, and for some odd reason, vertically oriented. Picture a refrigerator box (my preferred target) with a nice big oval pattern on it, wide in the middle, narrower toward the top and bottom, extending almost from side-to-side on the box, and almost from top to bottom. That's at 7 YARDS! Buckshot patterns tighter than the 7 1/2s do, maybe by six inches in all dimensions.
Needless to say, I don't think you want to come wanderin' around my house with ill intent, unless of course you happen to be totin' something bigger than a 12-bore! :haha:
So, back to the question about shot in a rifled muzzleloader. My contention is that it will work, won't hurt anything, and could be lots of fun. But you'd better be very close to your rabbit target to have enough pattern density to make a humane kill.