You posed a tough question, but one that I also asked myself when I first got into black powder... and I ended up with three .50 caliber rifles, a flinter long rifle and 2 cap-locks, one cap-lock for target shooting and the other, lighter, cap-lock with a shorter-barrel for deer hunting.
Initially, I thought strongly about a .36 caliber for paper punching. The supplies are less expensive since the round balls don't have as much lead in them... and buckshot (which can be fired through .36 caliber rifles) is less expensive per "ball" than even rifle balls you cast yourself.
The smaller .36 caliber also doesn't require as much powder as the larger calibers to push the ball to higher velocities for longer range shooting, but the pint-sized .36 caliber round ball runs outta "steam" at shorter ranges than the larger caliber rifle balls and the wind definitely effects the smaller, lighter rifle balls much more.
Since I intended to hunt deer, I needed "enough" caliber to get the job done well... and the .50 caliber fulfills that task quite well. The .50 caliber will also give the gong a good solid ring at the longer rangers (out to 100 yards or possibly further).
However, the cost of the .50 caliber rifle balls and the larger amount of powder one must use is evident which may or may not be a big consideration to you, but it is to a good many of us.
Also, the greater availability of supplies for the .50 caliber seems evident to me, but if you have a good firearms dealer nearby who also has a good black powder department, that shouldn't be a worry.
Once you get into black powder shooting and make friends who also love to shoot "smoke-poles", you may find that you'd enjoy getting out in the woods and doing some deer hunting with your friends.
This is where the .50 caliber would definitely be your best choice as compared to a .45 or a .40 caliber if .40 caliber rifles are even legal in your State for deer.
Since there is very little difference in the cost of supplies or the amount of recoil with either a .45 or a .50 caliber, I'd recommend the .50 caliber and never look back.
However, all of that said, the final decision is totally up to you. Decide what you prefer... and then, go-for-it! :v
Make "good" smoke... :thumbsup:
Strength & Honor...
Ron T.