Dixie is currently offering a gun just like it. Theirs is made by Euro Arms and their asking $725 in their 2005 catalog.
I don't know where you can get a spring, but maybe Dixie Gunworks could help you. They sometimes can do a pretty good job of matching the part if you make arrangements with them and send them the pieces.
In
CONFEDERATE ARMS , William A Albaugh III & Edward N Simmons, 1957 it shows one of these guns in plate 105, with the following description:
J.P.Murray artillery carbine. 40" overall, 23 1/2" barrel, .58 caliber, brass buttplate, trigger guard, flat band and double front band. Notched rear sight. Iron swivels on front band and front of trigger guard. Lockplate stamped "J. P. Murray, Columbus, Ga" Barrel breech stamped "PR., F.C.H."...Made at the Greenwood & Gray factory where Murray was Master Armorer.
In the back of the book under MURRAY, JOHN P.-COLUMBUS, Ga, the book goes on to say the letters F.C.H. were the initials of the inspecting officer F.C.Humphries.
It goes on to say Murray was an Englishman who lived in Charleston, S.C. until his death in 1910.
The Dixie Catalog says: "Originally manufactured by Eldridge S. Greenwood and William C. Gray of Columbus, Georgia, between 1862 and 1864 for Confederate Artillery units during the Civil War. Since raw materials were difficult to obtain in the South, Southern citizens donated their personal brass items to be melted down to manufacture some of the hardware for these guns. N/SSA approved."
There is only one size of musket cap so if the ones you have are loose, it may be because the nipple is worn, or because the caps you have are a little on the large size.
The Dixie gun uses the standard 5/16-24 nipple (Dixie # NP1335).