Emacs 27.1 daemon segfaulting when loading theme as emacsclient connects

Question

Emacs 27.1 from:

1. Homebrew/Linuxbrew, compiled from source, on either RHEL 6.10 or Ubuntu 16.04 Xenial.
2. EVM, using pre-compiled Travis binaries on Ubuntu 16.04 Xenial.

crashes for me with a segmentation fault when connecting a second terminal-based emacsclient:

Starting Emacs daemon.
Fatal error 11: Segmentation fault
Backtrace:
/tmp/tmp/emacs-27.1-travis-linux-xenial/bin/emacs[0x4aacc9]
/tmp/tmp/emacs-27.1-travis-linux-xenial/bin/emacs[0x408f6b]
/tmp/tmp/emacs-27.1-travis-linux-xenial/bin/emacs[0x4a959e]
/tmp/tmp/emacs-27.1-travis-linux-xenial/bin/emacs[0x4a9933]
/tmp/tmp/emacs-27.1-travis-linux-xenial/bin/emacs[0x4a9970]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x11390)[0x7f77c9db6390]
Segmentation fault

I have tried building with debug symbols and the stack trace was no more informative.

Sometimes it seems to take more variations/clients connecting before it crashes but a reliable reproduction is:

1. Start emacs --fg-daemon.
2. Connect with emacsclient -t.
3. Suspend the first client.
4. Connect with another emacsclient -t.
5. Server crashes.

I have narrowed it down to theme loading, with this .emacs being sufficient:

(defun my/after-make-frame-functions-hook (frame)
  (with-selected-frame frame
    (load-theme 'tango-dark t)))

(if (daemonp)
    (add-hook 'after-make-frame-functions 'my/after-make-frame-functions-hook))

I have also tried this:

1. with emacs -Q -l ~/.emacs --fg-daemon to rule out any site or other init files.
2. with multiple different themes.
3. with --bg-daemon (more likely to mess up your console - it is easier to use --fg-daemon and test clients from a different terminal).
4. using the new server-after-make-frame-hook instead of after-make-frame-functions.
5. without load-theme but still running a hook/same structure - fixes the problem.
6. using (server-start) from inside an existing Emacs process - FIXES the problem.

Does anybody else see this? Am I doing anything obviously wrong here? The code has worked for multiple major releases. What is the next thing to try?

EDIT

The same problem exists in the emacs-27 branch (0407b15500) and master (5824c209ba). The GDB stack trace of master is more informative:

Program received signal SIGSEGV, Segmentation fault.
0x0000000000000000 in ?? ()
(gdb) bt
#0  0x0000000000000000 in ?? ()
#1  0x0000000000490e45 in load_color2 (f=f@entry=0xa1d440, face=face@entry=0xb1e930, name=XIL(0x7ffff4d337dc), target_index=target_index@entry=LFACE_FOREGROUND_INDEX, color=color@entry=0x7fffffff80c0) at xfaces.c:1195
#2  0x0000000000491a6c in load_color (target_index=LFACE_FOREGROUND_INDEX, name=<optimised out>, face=0xb1e930, f=0xa1d440) at xfaces.c:1258
#3  map_tty_color (f=f@entry=0xa1d440, face=face@entry=0xb1e930, idx=idx@entry=LFACE_FOREGROUND_INDEX, defaulted=<optimised out>) at xfaces.c:6185
#4  0x0000000000491db7 in realize_tty_face (cache=0xa96c20, attrs=0x7fffffff8130) at xfaces.c:6259
#5  realize_face (cache=cache@entry=0xa96c20, attrs=attrs@entry=0x7fffffff8130, former_face_id=former_face_id@entry=0) at xfaces.c:5829
#6  0x00000000004933be in realize_default_face (f=<optimised out>) at xfaces.c:5745
#7  realize_basic_faces (f=f@entry=0xa1d440) at xfaces.c:5603
#8  0x0000000000495ae0 in update_face_from_frame_parameter (f=f@entry=0xa1d440, param=param@entry=XIL(0x2340), new_value=new_value@entry=XIL(0x7ffff4d337fc)) at xfaces.c:3669
#9  0x0000000000419773 in Fmodify_frame_parameters (frame=<optimised out>, alist=<optimised out>) at frame.c:3353
#10 0x000000000050ff53 in Ffuncall (nargs=3, args=args@entry=0x7fffffff8300) at eval.c:2806
#11 0x0000000000546958 in exec_byte_code (bytestr=XIL(0x7ffff4cbc72c), vector=XIL(0x7ffff4cbc6cd), maxdepth=<optimised out>, args_template=<optimised out>, nargs=nargs@entry=140737300383440, args=<optimised out>, args@entry=0x18) at bytecode.c:632
#12 0x000000000050fc11 in fetch_and_exec_byte_code (args=0x18, nargs=140737300383440, syms_left=<optimised out>, fun=XIL(0x7ffff4cbc6d0)) at eval.c:2928
#13 funcall_lambda (fun=XIL(0x7ffff4cbc6d0), nargs=nargs@entry=3, arg_vector=0x18, arg_vector@entry=0x7fffffff8488) at eval.c:3009
#14 0x000000000050fe9f in Ffuncall (nargs=4, args=args@entry=0x7fffffff8480) at eval.c:2820
#15 0x0000000000546958 in exec_byte_code (bytestr=XIL(0x7ffff4d43e34), vector=XIL(0x7ffff4d43ab5), maxdepth=<optimised out>, args_template=<optimised out>, nargs=nargs@entry=140737300937400, args=<optimised out>, args@entry=0x17) at bytecode.c:632
#16 0x000000000050fc11 in fetch_and_exec_byte_code (args=0x17, nargs=140737300937400, syms_left=<optimised out>, fun=XIL(0x7ffff4d43ab8)) at eval.c:2928
#17 funcall_lambda (fun=XIL(0x7ffff4d43ab8), nargs=nargs@entry=1, arg_vector=0x17, arg_vector@entry=0x7fffffff86b8) at eval.c:3009
#18 0x000000000050fe9f in Ffuncall (nargs=2, args=args@entry=0x7fffffff86b0) at eval.c:2820
#19 0x0000000000546958 in exec_byte_code (bytestr=XIL(0x7ffff4e983ec), vector=XIL(0x7ffff4d43805), maxdepth=<optimised out>, args_template=<optimised out>, nargs=nargs@entry=2, args=<optimised out>, args@entry=0x16) at bytecode.c:632
#20 0x000000000050fc11 in fetch_and_exec_byte_code (args=0x16, nargs=2, syms_left=<optimised out>, fun=XIL(0x7ffff4d43808)) at eval.c:2928
#21 funcall_lambda (fun=XIL(0x7ffff4d43808), fun@entry=<error reading variable: Cannot access memory at address 0x308>, nargs=nargs@entry=140737488324816, arg_vector=0x16, arg_vector@entry=0x2) at eval.c:3009
#22 0x000000000051278c in apply_lambda (fun=<error reading variable: Cannot access memory at address 0x308>, args=<optimised out>, count=<error reading variable: Cannot access memory at address 0x300>) at eval.c:2953
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Lisp Backtrace:
"modify-frame-parameters" (0xffff8308)
"set-frame-parameter" (0xffff8488)
"disable-theme" (0xffff86b8)
"load-theme" (0xffff88d0)
"progn" (0xffff8a18)
"unwind-protect" (0xffff8ad8)
"let" (0xffff8bf8)
"my/after-make-frame-functions-hook" (0xffff8e70)
"run-hook-with-args" (0xffff8e68)
"make-frame" (0xffff9130)
"server-create-tty-frame" (0xffff93c8)
"server-process-filter" (0xffff9948)

Answer 1

This is a bug in Emacs. The current workaround is not to load a theme with a suspended frame. This can often happen if you use daemon mode with init code like that shown. Theme-loading code of that style can be adjusted to not run more than once. For example, see later answers to this question.

There is a patch in the emacs-27 branch already but obviously it is not released yet. The patch currently only prevents a crash; it errors when you try to load a theme with a suspended frame.

EDIT

The patch has been modified to allow loading themes with suspended frames.