Why does local-key-binding return a 1 for a key with no local binding

Question

When I evaluate (local-key-binding key) for any arbitrary key that does not have a local binding, I expect a return value of nil, but I almost always get a 1. If the key actually does have a local binding (say from a major mode) in the current buffer, then local-key-binding behaves as expected and returns the function to which the key is bound.

My question then, is why does local-key-binding often return 1 when there is in fact no local binding for the given key?

As an example, after opening emacs with emacs -Q, type

(key-binding (kbd "C-h h"))
(local-key-binding (kbd "C-h h"))

in the scratch buffer and evaluate each of the forms. Notice that key-binding returns view-hello-file, and local-key-binding return 1. The binding for C-h h is defined globally, not locally, so I expect local-key-binding to return nil. Now try

(local-set-key (kbd "C-h h") 'describe-bindings)
(key-binding (kbd "C-h h"))
(local-key-binding (kbd "C-h h"))

and evaluate each of the forms. Notice that both key-binding and local-key-binding return describe-bindings. Now try

(local-unset-key (kbd "C-h h"))
(key-binding (kbd "C-h h"))
(local-key-binding (kbd "C-h h"))

and evaluate each of the forms. Notice that key-binding has gone back to view-hello-file, and local-key-binding now shows nil.

Answer 1

local-key-binding uses lookup-key whose doc-string contains this:

A number as value means KEY is "too long"; that is, characters or symbols in it except for the last one fail to be a valid sequence of prefix characters in KEYMAP. The number is how many characters at the front of KEY it takes to reach a non-prefix key.

local-unset-key does not reverse the effect of local-key-binding, because it does not actually do what its doc-string claims it does:

Remove local binding of KEY

But what actually happens boils down to:

(define-key (current-local-map) "\C-hh" nil)

(One could argue that it does what the doc-string claims, but it certainly doesn't do what one would expect when first reading that doc-string).

The keymap went through these stages:

(keymap)
(keymap (8 keymap (104 . describe-bindings))))
(keymap (8 keymap (104)))

To remove the binding for a key instead of re-binding it to nil you can use kmu-remove-key from my library keymap-utils.

When a key is bound to nil in a local keymap that prevents key lookup to consider the binding in the parent keymap, but it does not override the global binding.