Of course. The canonical way to do that is to use suppress-keymap on your mode's keymap. E.g.
(let ((map (make-keymap)))
In the unlikely case that you want to keep the self-inserting behavior of non-digits, then you should instead manually bind each digit to digit-argument as you ...