Some many years ago my dad supervised a work camp occupied by inmates of the state system. The inmates brought a fawn in one spring thinking it abandoned. No way of knowing if it was or not, but point of story is that "Bucky" became a permanent resident of the camp. After a few years Bucky became a pretty respectable sized buck. During the rut he attacked a woman visiting the camp! No way of knowing why, but it was clear that he could not be trusted around the camp any longer. The inmates painted his antlers red and transported him many miles away to an area they were working in. Was never seen again and there were no reports of red antlered deer being killed that year.
Point is, if they are hanging around fearlessly they could become a problem. I would shoot em and eat em, but if that bothers you, make it a point to run them off with plenty of noise etc. until they no longer feel comfortable.