Looking for a better way to test whether the current Emacs process is running under GNU Screen

Question

NB: For the sake of this question, please assume that the underlying OS is some flavor of Unix. If more specificity is required, then please assume that it is either Darwin or some flavor of Linux.

---

In my .emacs file, I would like to map a particular key combination only when the current emacs process is a descendant of a screen (GNU screen) process. I am looking for a robust way to test for this condition.

I know that (emacs-pid) will evaluate to an integer corresponding to the pid of the current emacs process.

Beyond this point, the only thing I can come up with is the following (very fragile, IMO) monstrosity:

(if (= 0
       (shell-command
        (format "pstree -pu $USER | grep -qP -- '-screen\\(\\d+\\)-.*-emacs\\(%d\\)'"
                (emacs-pid))))
    (global-set-key ...)
)

Basically, this code runs a global-set-key command if the following expression evaluates to 0:

(shell-command
 (format "pstree -pu $USER | grep -qP -- '-screen\\(\\d+\\)-.*-emacs\\(%d\\)'"
         (emacs-pid)))

To evaluate this expression, Emacs will run the command below¹ in a subshell, and will use its exit status as the expression's value:

pstree -pu $USER | grep -qP -- '-screen\(\d+\)-.*-emacs\(<EMACS_PID>\)'

This shell command, in turn, will have an exit status of 0 if and only if the line in the output of pstree -pu $USER that contains the substring -emacs(<EMACS_PID>) is preceded by a substring that matches the Perl regex -screen\(\d+\)-.

In my experience, Emacs extension code that relies on parsing the output of some shell command is always extremely fragile.

Therefore, I am looking for an alternative way to solve this problem that does not rely on shell-command or similar, but rather directly interrogates the ancestors of (emacs-pid) (using the underlying OS's getppid function somehow). This would entail traversing the sequence of (emacs-pid)'s ancestors until finding either one whose associated command is screen² or one whose pid is 1 (i.e. it's the root process).

---

^{1 Here and below, I use the expression <EMACS_PID> as a placeholder for the integer (emacs-pid).}

^{2 Admittedly, the problem of determining, in a robust way, that a particular PID corresponds to a GNU-screen process may already be pretty non-trivial.}

Answer 1

Skip all the pid stuff, and just check the STY environment variable. If it is set, then emacs is running inside screen.

Answer 2

@db48x's answer is undoubtedly the best answer to the question. But it depends on the good graces of screen which conveniently sets the STY variable that sub-processes can check (tmux sets a TMUX variable, so it's equally cooperative). What if you have a less cooperative process that does not set a variable? Doing the "pid stuff", i.e. walking up the tree of processes until you find either pid 1 or the process of interest can easily be done within emacs: you don't need to parse shell command output. Here is a simple implementation:

(defun ndk/walk-up-process-tree-up-to (name)
  (catch 'found 
    (let ((pid (emacs-pid)))
      (while t
        (let* ((attrs (process-attributes pid))
               (ppid (cdr (assoc 'ppid attrs)))
               (comm (cdr (assoc 'comm attrs))))
          (cond ((string= comm name)
                 (throw 'found pid))
                ((= pid 1)
                 (throw 'found nil))
                (t (setq pid ppid))))))))

The function returns the pid of the (first) process it finds walking up the process trees with the given name, or nil if no such name is found.

I tested it with an emacs started from screen, which was started from an xterm, which was started from my GUI emacs. The process tree as viewed with pstree looked like this:

$ pstree -psl $$
systemd(1)───systemd(2728)───gnome-shell(2949)───emacs(622022)───bash(623167)───xterm(796197)───bash(796199)───screen(796203)───screen(796204)───bash(796205)───emacs(796207)───bash(796226)───pstree(796325)

Evaluating (ndk/walk-up-process-tree-up-to "screen") in the child emacs whose pid is 796207 returns 796204, the pid of the child screen, the closest ancestor of the child emacs.

A couple of weaknesses: 1) As noted above, it returns the pid of the closest ancestor, so if you want the top-level screen e.g. in the above example, the function needs modifications. 2) A more significant weakness might be that the comm string does not match naive expectations (although it did in this case): a substring check might be better than the equality check I used. 3) It probably does not work on non-Unixy systems, since I doubt that they have a pid 1.