DC,
You do not mention where you live, so it is hard to answer your question. As you can see from the answers, attitudes are varied between the different states.
You must consider laws, customs and friendships before deciding how to act in this.
As an example, in Louisiana, you are required to make an effort to retrieve game and property owners cannot prevent you from crossing boundaries and fences to do so. You can be required by the owner to leave your gun while doing so in order to keep you from hunting on their land without permission.
I hunted in a pretty lawless area for a while and we had problems with a family of poachers down the road.
(Wondered how we got rights to prime hunting land for a dollar a year for 5 years!) We also had some good neighbors. We all watched for the poachers and made deals with the other neighbors regarding the rules for recovering game while still armed so that we all knew where we stood. None of us wanted to be in the woods unarmed with the poachers around.
We also developed the habit of watching for the other guys' properties and reporting to each other. Made for good friendships.
In time, the poachers were caught a few times and that problem went away. They were also stared down a few times.
I will also point out that the national standard for ownership of wounded game is the rule of first blood. The first to draw blood on game owns it and should recover it. This may not be the case in a few states, but is the generally accepted custom.
I suggest that you attend a hunter safety course in your state. They will clearly address these questions for your area.
YMHS,
CrackStock