Why the inconsistent behaviour of Shift in keybindings?

Question

Let's say I want to bind C-Z in a keymap like this:

(define-key d-emacs-mode-map (kbd "C-Z") #'d-emacs-test)

Oh no, now I've rebound the lowercase-variant C-z as well! I guess what I have to do is

(define-key d-emacs-mode-map (kbd "C-S-z") #'d-emacs-test)

Alright, now things seem to work as they should: Holding Ctrl and Shift and tapping z triggers d-emacs-test while holding Ctrl and tapping z triggers the original binding on C-z, say d-emacs-test2. But if I now bind

(define-key d-emacs-mode-map (kbd "M-S-z") #'d-emacs-test)

and then hold Meta and Shift while tapping z, d-emacs-test2 is triggered, whereas if I bind

(define-key d-emacs-mode-map (kbd "M-Z") #'d-emacs-test)

and do the same, d-emacs-test is triggered. If I now do C-h k, hold Ctrl, Meta and Shift and tap z, I get

C-M-S-z is undefined

Super, Hyper and Alt seem to work similarly to Meta, so really it seems an uppercase binding needs the shift-modifier specified as a modifier if and only if it contains a Ctrl modifier while otherwise it needs the letter written in its upcased form. This was tested on Emacs 31.0.50.

What causes this inconsistent behavior?

Answer 1

The reason for the inconsistencies is that key bindings are partly based on keys and partly based on characters. Is Mod+Shift+A the modifiers Mod and Shift applied to the key A, or the modifiers Mod and Shift applied to the character a (which is what you get when you press the key A with no modifiers), or the modifier Mod applied to the character A? It depends…

Shift and Ctrl are peculiar because they can be applied to characters as well as to keys. For Shift and letters, it's obvious: Shift applied to a letter changes the case of the letter¹. Both keyboard input and the kbd macro transform Shift+letter into the corresponding uppercase letter if applicable. For Ctrl, there are characters known as control codes: they're unprintable in the sense that they don't usually have associated glyphs, but they can be encoded and typed. There are control characters associated with each of the 26 Latin letters as well as with the punctuation characters @[\]^_?. Both keyboard input and the kbd macro transform Ctrl+key into the corresponding control character if applicable.

The kbd macro adds another layer of complexity because it's meant to “do what I mean” in simple cases, but it's unintuitive in more delicate cases. For example, (kbd "C-z") is the same thing as (kbd "C-Z"), but (kbd "M-z") is not the same thing as (kbd "M-Z"). I recommend to avoid kbd and instead use the event representation used in key bindings. You can use the string-based syntax if you want, but you don't have to: you can use the [(modifier… base-key)] syntax for everything. With function keys, kbd is just a more compact version: (kbd "C-<left>") is [(control left)]². With character-based events, the syntax is [(modifier… ?c)] where c is the desired character (? in Lisp syntax introduces a character literal).

(define-key d-emacs-mode-map [(control ?z)] 'control-z)
(define-key d-emacs-mode-map [(control shift ?z)] 'control-shift-z)
(define-key d-emacs-mode-map [(meta ?z)] 'meta-z)
(define-key d-emacs-mode-map [(meta shift ?z)] 'meta-shift-z)
(define-key d-emacs-mode-map [(control meta ?z)] 'control-meta-z)
(define-key d-emacs-mode-map [(control meta shift ?z)] 'control-meta-shift-z)

In an ideal world, there would be no association between a small set of unprintable characters and certain printable characters with the control modifier. But this association was historically built into text-oriented terminal hardware and it has permetated how keyboard input is represented in Emacs and many other applications that have a history with text terminals (and even many that don't). In addition to event representation and the kbd macro, the association is reflected in key maps all over the place, where "\C-z" is meant to represent Ctrl+Z and not some unprintable, untypeable character.

On a related note, the existence of control characters also strikes with bindings of certain function keys which, on certain terminals, send a control character. In Emacs, this permeates through predefined translations, printed representations and key maps. For example, "\t" in a key map is meant to represent the Tab key, and Emacs shows it as TAB, but that is the same character as "\C-i", so Emacs translates the Tab key to the character "\C-i". See How to bind C-i as different from TAB? on that topic.

¹ _{Note that this does not have to correspond to the actual effect of holding Shift while pressing the key. It almost always does, but that's a matter of convention, not a technical constraint. On a French keyboard, holding Shift while pressing accented letters does not insert the capital letter, but a digit or punctuation character.}

² _{You can even abbreviate [(control left)] as [C-left].}