Solved it: it was because Emacs was installed as a flatpak. For some reason, this mean the Emacs shell had many differences. I wasn't able to cd into certain folders. Other folders didn't exist: /usr/local, for example.
After uninstalling, and re-installing from the Ubuntu repository, everything works as expected.
I hope this can be helpful to someone. My ...
You can redefine ffap-file-finder. This is normally set to find-file, which doesn't do env variable expansion. So define a function like find-file that does do env variable expansion:
(defun find-file-env (fname)
(find-file (substitute-in-file-name fname)))
and assign it to ffap-file-finder:
(setq ffap-file-finder #'find-file-env)
An interesting question. It's not configurable at all, so I recommend looking at pcomplete-insert-entry. This is where the completed text is inserted, and it's what calls comint-quote-filename. Note that it handles some fun edge cases that you'll also have to handle. In particular, if there's already a backslash in the buffer then it handles the insertion a ...
Figured it out: (setq shell-input-autoexpand nil). I spent a long while looking at comint-input-autoexpand which was getting mysteriously set to 'history: this is done by shell-mode based on shell-input-autoexpand. I don't see an easy way to make it prompt-dependent, though.
In Shell mode (and many other modes that run an external command interpreter), press C-c C-c (comint-interrupt-subjob). That's the C-c prefix of mode-specific commmands, followed by C-c which is chosen to be like C-c in a plain terminal.