Every muzzleloading fan ought to have a couple of brass or bronze punches, for tasks such as removing/installing sights, same for wedge keys, etc. Also used for adjusting sights for windage.
Brass or bronze are preferable to steel as being softer than steel they will not damage steel parts. Buy them, or make your own. Brass stock is available at major hardware stores. Small sizes of bronze stock may be had from your local welder. Search internet for mail order suppliers. Last time I needed some bronze welding rod to make some flint nibblers I took in a bag of elk jerky. Easy trade.
When filing brass or bronze stock, be sure to clean files frequently. Any type of punch or drift will work better if the workpiece is well supported. For example, lay your barrel on its side on a sturdy table or bench padded with several layers of newspaper to protect the finish. Freehand or across the lap just leads to frustration.
SAL