run a single ERT test programmatically

Question

I'm trying to write a piece of code that runs a single ERT test programmatically. In the ERT manual, they show an example that creates a test body and then runs the body, like so:

(ert-deftest ert-test-record-backtrace ()
   (let ((test (make-ert-test :body (lambda () (ert-fail "foo")))))
    (let ((result (ert-run-test test)))

But... (ert-run-test won't work with a normal ert-deftest name such as the one shown above, ert-test-record-backtrace. If I try doing that I see an error

(ert-run-test 'ert-test-record-backtrace)
Debugger entered--Lisp error: (wrong-type-argument arrayp my-deftest)

The code (ert-run-tests-interactively) runs the usual ERT function that prompts for tests in the minibuffer.

How can I run one or more ERT tests programmatically? Is it even possible? I'm working on this kind of syntax right now, but it is not working. I think I need to extract the function body value of my-deftest, but I don't know how to do that yet.

(ert-run-test (make-ert-test :body my-deftest))

Update: I tried to put my whole test function inside the make-ert-test format shown in the example above, and was able to run the test. Not pretty code, because of the indenting. And the result was in square brackets, which I don't understand yet. But I can see the value: t in there that I would like to extract. So I'm getting closer.

(setq result
  (ert-run-test
   (make-ert-test :body
                  (lambda ()
                    ;; my ert test code here
                    (should (equal tcoll output))
                    ))))

[cl-struct-ert-test-passed "" (((should ...) :form (equal ... ...) :value t :explanation nil))]

After more experimentation to dig the answer out of the result returned (it was an array), here is the code that gets me to the t that I could see.

(setq data (aref result 2))
(((should (equal tcoll output)) :form (equal ...)) :value t :explanation nil))
(nth 4 (car data))
t

It works, but it seems like a brute force approach because I have to copy my ert-deftest code into a separate lambda function as shown above.

Answer 1

Let's use the example from the ERT manual:

(ert-deftest ert-test-record-backtrace ()
  (let ((test (make-ert-test :body (lambda () (ert-pass "foo")))))
    (let ((result (ert-run-test test)))
      (should (ert-test-failed-p result)))))

The test should pass obviously. A list of symbols naming an ert-test object will be returned with

(apropos-internal "" #'ert-test-boundp) ;⇒ (ert-test-record-backtrace)

And finally show the number of available tests that passed as expected:

(ert--stats-passed-expected (ert-run-tests 't (lambda (&rest args)))) ;⇒ 1

The listener in ert-run-tests may be an arbitrary function called for side-effects.

Answer 2

You can try ert-run-tests-batch, e.g.

(ert-run-tests-batch 'ert-test-record-backtrace)

but the results are not returned as a value, but rather via messages in *Messages* (or on stdout). Otherwise, you can use ert-run-tests which is a bit more complex to use because you need to provide a listener function:

(ert-run-tests 'ert-test-record-backtrace
               (lambda (event &rest args)
                 (message "Test event %S: %S" event args)))