I think this is an interesting thing to think about. I'm always debating with myself as to which pistol to bring when and with what rifle. I am very fond of my Remington Navy and 1860 Colt for sidearms when I hunt. The Remington Navy is a great grouse-getter, and usually my go-to sidearm. As I hunt with either a .75" musket, or a .62 or .58" rifle, a handgun in the same caliber isn't real practical, although there are more pistols in those calibers, than in .32". But usually I wouldn't want to pack that big of a pistol.
I do carry a Plain's Pistol in .50", heavily loaded, when calling in bear and cougar, so I can make a close shot to my far right, when something comes in behind me, as it is hard to swing hard right with a rifle when sitting. But with a .32" I assume you are after much smaller game.
As a companion to a .32" rifle, that does not leave many options. A single shot in .32 would be very sweet, but one would almost have to have a barrel custom made, or the whole pistol, or put out the $$$$ for a Le Paige and re-line the barrel to .32".
The baby dragoons would be pretty close in caliber, but are short barreled pistols. But, depending on what size ball your rifle shoots, might be a match, if you wanted that small of a pistol.
With such a small bore rifle, perhaps a large bore pistol would be a better choice. If you want to stay somewhat small bore, the Colt Navy and Remington Navy pistols are very sweet, and a .36" has a better wallop than most people think. Also, the smaller versions such as the 1862 "Pocket Police" are nice accurate pistols, are .36 calibre, and.....smaller and lighter if that's important.
On the other hand...a good Colt or Remington in .44 is a great sidearm. I prefer the 5.5 and 6" barreled versions over the 7.5 and 8" long-toms. Again, with a "small" rifle, perhaps a large bore pistol is a better "match". ???
Or...perhaps re-barrel or rebore the rifle to .36", and have a ton of choices in something that shoots the same size ball. !!!! But you'd have to be very careful to get your bore sized where it shot the standard .375" ball, or you'd be right back where you started from.
As far as handguns in a different ball size from the rifle, you can still share powder and caps, or flints, and patches between the two. It's a "cool factor" for sure, to have them in the same caliber, but certainly not important, or necessary. But necessary is not what this hobby is about for sure.
Good luck, I love over-thinking this stuff.