We have a job running in CI that verifies the output of an ert batch run. The job also includes examples of failing tests which produce a backtrace. The job tests if the error message output matches our expectation.
This has worked well with Emacs 28.2 with a simple call to diff, checking if the error message is exactly the same. But after upgrading to Emacs 29.3 the error message changes between runs.
Emacs 28.2:
Loading /solution/example-syntax-error.el (source)...
Debugger entered--Lisp error: (void-variable DEF&*@)
eval-buffer(#<buffer > nil "/solution/exampl..." nil t) ; Reading at buffer position 41
load-with-code-conversion("/solution/exampl..." "/solution/exampl..." nil nil)
load("/solution/exampl..." nil nil t)
load-file("example-syntax-error.el")
eval-buffer(#<buffer > nil "/solution/exampl..." nil t) ; Reading at buffer position 151
load-with-code-conversion("/solution/exampl..." "/solution/exampl..." nil t)
load("/solution/exampl..." nil t)
command-line-1(("-l" "ert" "-l" "/solution/exampl..." "-f" "ert-run-tests-batch-and-exit"))
command-line()
normal-top-level()
Emacs 29.3:
Loading /solution/example-syntax-error.el (source)...
Error: void-variable (DEF&*@)
mapbacktrace(#f(compiled-function (evald func args flags) #<bytecode -0x77b476f04510f1b>)) <--- the bytecode output changes
debug-early-backtrace()
debug-early(error (void-variable DEF&*@))
eval-buffer(#<buffer > nil "/solution/example-syntax-error.el" nil t)
load-with-code-conversion("/solution/example-syntax-error.el" "/solution/example-syntax-error.el" nil nil)
load("/solution/example-syntax-error.el" nil nil t)
load-file("example-syntax-error.el")
eval-buffer(#<buffer > nil "/solution/example-syntax-error-test.el" nil t)
load-with-code-conversion("/solution/example-syntax-error-test.el" "/solution/example-syntax-error-test.el" nil t)
load("/solution/example-syntax-error-test.el" nil t)
command-line-1(("-l" "ert" "-l" "/solution/example-syntax-error-test.el" "-f" "ert-run-tests-batch-and-exit"))
command-line()
normal-top-level()
Symbol's value as variable is void: DEF&*@
I've traced back the problem to the introduction of debug-early.el, which is now always used in batch mode. This lambda changes the output every run of the #<bytecode -0x77b476f04510f1b> part, line 3 of the quote above.
So far I've only found the backtrace-on-error-noninteractive variable to deactivate the backtrace, which is not what we want because we don't set that flag in production, making the test less useful.
I've also tried to set the variable debugger, but that doesn't seem to have an effect. (I later saw that it's already set to debug in batch mode, so relevant code doesn't seem to care about that variable.)
Is there any way to make the output the same on every run again?
