Early in my hunting career on a fine, still, frosty morning a group of does surprised me by slowly moving into my area from behind. They did not know I was there, until I set the trigger on the TC Renegade. Stopped them dead in their tracks and they never came into my shooting lane after that. So, yes, it certainly can spook them.
The unset trigger on that rifle is quite stiff and the first time I took a shot a game, an 8 point, without setting the trigger the shot pulled to the front shoulder/neck junction. It severed his windpipe, so was fatal as he ran down the slope. But none of blood on that Indiana buck leaked out of his veins and the meat tasted like field corn. Not very appetizing. I learned to shoot more accurately with that rifle without setting the trigger, but if the opportunity allows I set it more often than not. A little wind or leaves crunching under their feet is adequate to prevent them from hearing it.
I do hold the trigger, cock the hammer and then release the trigger when ready to shoot when hunting. It's not hard to do safely. Once full cocking and loosing an chance was a lesson learned. Unnatural noises get their unwanted attention fast.