I would have suggested a .454" ball as well, especially since pure lead will cast a tad smaller than "impure," i.e., harder alloy. Also, be aware that not all bullet molds, be they roundball, conical, or gas checked (for center fire arms) cast to their marked dimension.* E.g., I have a roundball mold, made by a very reputable manufacturer, marked .451", which didn't shave a ring of lead from my Rem. 1858 Army either. When I cast another batch (pure lead) and measured them with a micrometer, they were only .448" - .449", so it's no wonder that they didn't shave a ring of lead when seated.
*slightly bigger is better than the reverse