0

When running ert tests, the seem to be run by default in string< order. For instance running tests

(require 'ert)
(ert-deftest t1111 () (print 'AAAA))
(ert-deftest t3333 () (print 'BBBB))
(ert-deftest t2222 () (print 'CCCC))
(ert-run-tests-batch :new)

will print

;; Actual output              ;; Wanted output

AAAA                          AAAA

   passed  1/3  t1111            passed 1/3  t1111

CCCC                          BBBB

   passed  2/3  t2222            passed 2/3  t3333

BBBB                          CCCC

   passed  3/3  t3333            passed 3/3  t2222

For interpretation it would be helpful to guarantee the order of the tests. How can this be achieved?

0

As far as I could tell, it is not possible to cause the selectors t (universe) or :new to run tests in the order of definition. The (member TEST1 TEST2 ...) selector however executes tests in the order specified.

For my tests I am therefore using one of two workarounds (both written ad-hoc and not tested for balanced parentheses):

1. Wrapper, that records the tests in a list.

(defconst NAMESPACE-test-list nil)
(defmacro NAMESPACE-deftest (name &rest args)
  `(progn
     (ert-deftest ,name ,@args)
     (setq NAMESPACE-test-list (nconc NAMESPACE-test-list (list ',name)))))

;; TEST DEFINITIONS

(ert `(member ,@NAMESPACE-test-list))

2. Scan file for ert-deftest with regular expression search.

;; TEST DEFINITIONS

(ert (cons 'member
       (with-temp-buffer
         (insert-file load-file-name)
         (cl-loop
           while (search-forward-regexp (rx bol "(ert-deftest" (+ blank)
                                            (group-n 1 symbol-start (*? nonl) symbol-end)))
                                        nil t)
           collect (intern (match-string 1))))))

3. Numbered test names

Alternatively, the alphabetic order of the tests can be (ab?)used, e.g.

(ert-deftest NAMESPACE-001:some-function () ...)
(ert-deftest NAMESPACE-002:other-function () ...)
...
(ert :new)

or

(defvar NAMESPACE-test-counter 0)
(defmacro NAMESPACE-deftest (name &rest args)
  `(ert-deftest ,(intern (format "NAMESPACE-%03d:%s" 
                           (cl-incf NAMESPACE-test-counter)
                           name))
      ,@args))
(NAMESPACE-deftest some-function () ...)
(NAMESPACE-deftest other-function () ...)

Both variants have serious downsides however:

  • The first variant causes maintenance issues. Want to reorder the test definitions? Now we also need to renumber all tests, which in turn confuses the version history.
  • The second variant breaks interactive lookup of the test definition from the M-x ert buffer.

If I have just missed a setting, that enables "execute in definition order", I'll be happy to hear of it however.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.