I saw a pair of Ultra High Tower pistols at a flea market that had been defarbed, the guy wanted $200 for the pair, they were in pretty good shape. He told me they Tryon pistols from Spain, I told him they weren't but he didn't believe me. I went home and looked them up just to be sure I was right. Whoever did the defarbing had left a hint of the stamp "Dixie Gun Works" on one of the barrels.
I think it depends on who is doing the defarbing and on what. Defarbe a TC and everyone will still know what it is, defarbe a tower pistol and the origin of the maker gets murky for most folk and drops the value.
The only thing I defarbed was a Green MT barrel that I struggled with the plug fit in my early builder days, the logo ended up on the top oblique flat when I had a perfect breech face fit. I had worked on it for hours and was tired of it, I didn't want to go a couple more flats so I punched out the lettering and drawfiled it off.
It is a personal thing, some one might think their TC Hawken is a real Hawken if they take the logo off, whatever floats your boat, I guess.
I think it devalues a factory gun, for the rank and file who don't know one maker from another like we do, they would probably want to know exactly what they are buying.