The ESC key has along with C- and M- in Emacs the special status of being a prefix key, so that the translation from key to key does not work.
Check out: How to remove bindings to the ESC prefix key and try:
(define-key key-translation-map (kbd "ESC") (kbd "`"))
( tested to work OK with Emacs 29 on Linux Mint 21 Cinnamon )
I assume that this above takes directly the keypress to re-map it instead of 'waiting' for a key code resulting from an interpreted keypress-combination.