FWIW I use a similar end block/jig/setup like you do, but I go outside to my deck railing and I use C-Clamps to hold the end block and a large ratchet clamp to hold the drill. That also makes getting rid of all the shavings and/or sanding dust easy too, where I'll just sweep it all off the deck ... no dust collector in my basement work shop, lol!... the hard part is holding the drill motor in my left hand and the rasp or gouge in my right hand.