A compilation-mode conundrum - matching and highlighting

Question

Whilst playing around with compilation-mode for running tests and using HIGHLIGHT faces for failed expectations I've encountered a bit of a conundrum.

Question: why does my compilation-errors-regexp-alist fail to match when the matching group/s are preceded by more than a certain number of characters. And how best to mitigate this given my matching requirements.

Let me try and demonstrate.

I'm using GNU Emacs 25.1.1 (x86_64-pc-linux-gnu, GTK+ Version 3.18.9) of 2017-01-14

Here's some minimal code that reproduces the issue:

(defface face-frame
  '((t :foreground "white" :background "black" :weight bold)) "frames out for a failing test")

(defface face-wrong
  '((t :foreground "white" :background "darkred" :weight bold)) "highlight incorrect output in red")

(defface face-expected
  '((t :foreground "white" :background "darkgreen" :weight bold)) "show expected output in green")

(defconst error-regexp-alist
  '(("^Received\n\\(\\(.\\|\n\\)+?\\)\nExpected\n\\(\\(.\\|\n\\)+?\\)\n\\(.*\\):\\([[:digit:]]+\\)\n--$"
     5 6 nil nil nil (0 'face-frame) (1 'face-wrong) (3 'face-expected))))

(define-derived-mode weird-compilation-mode compilation-mode "Fixture Compilation"
  (set (make-local-variable 'compilation-error-regexp-alist)
       error-regexp-alist))

(defun demonstrate-weirdness ()
  (interactive)
  (with-current-buffer (get-buffer-create "*weird-regexp-matching-issue*")
    (compilation-start "./fixture.sh" 'weird-compilation-mode (lambda (m) (buffer-name)))
    ))

To reproduce:

• Clone https://github.com/codeasone/compile-mode-weirdness.git
• cd compile-mode-weirdness
• Load compile-mode-weirdness.el
• Run M-x demonstrate-weirdness

A compilation-mode buffer should appear with the output of fixture.sh, which produces test output to match against.

Looks good to me and it's what I want; output Received in error is highlighted RED and Expected output is highlighted GREEN, for both error cases

BUT, add a single addition 1 (or any other character) to the start of fixture.sh (after the cat << HERE of course) then switch back to the *weird-regexp-matching-issue* buffer and hit g to (recompile)

I designed the regexp to group everything between Received and Expected including new-lines, and similarly between Expected and file:line information:

^Received\n\\(\\(.\\|\n\\)+?\\)\nExpected\n\\(\\(.\\|\n\\)+?\\)\n\\(.*\\):\\([[:digit:]]+\\)\n--$

Note: it's a shame Emacs doesn't support lookbehind and lookahead facilities as it would have made things easier...

It appears to work fine outside the context of compilation-mode. I've tested it using various input strings via regex-tool, helm-regex, and using (s-match ...) to check my own sanity.

Regular expressions can be such a wind up...

Why is the matching and highlighting failing when the "preamble" output is extended beyond a certain length?

Perhaps someone with more experience with Emacs regular expressions or compilation-mode internals can help me out. Hopefully the code + examples will help others who find themselves in a similar boat using compilation-mode in the future.

Answer 1

I think this is what's going on:

The shell command produces output and at the same time compile.el wants to put font-lock properties on it using a list of regexps. But how should it know, whether enough text has been produced in order to satisfy all regexp's requirements ? For example, there could be one matching the start and end of buffer, which would only succeed, if the complete output would be matched at once. But that's not how compile.el operates: It uses font-lock, which parses the output in batches as they arrive.

The same applies to your test case, albeit in a less severe way. We can see what's going on, if we trace the function compilation--ensure-parse using trace-function-background. Here is a partial result, when applied to the erroneous case (with an added 1).

======================================================================
1 -> (compilation--ensure-parse 184)
1 <- compilation--ensure-parse: nil
======================================================================
1 -> (compilation--ensure-parse 684)
1 <- compilation--ensure-parse: nil
======================================================================

Checking the position 684 in the compilation buffer, we see that it corresponds to the beginning of this sub-string.

/path/to/test:1234
--

And it becomes clear, why this doesn't work: Your regexp requires the text to end in a double-hyphen, but font-lock has stopped just two lines short of the first occurrence of this string, so the first part of the output does not match.

What you can do about it, is to make sure, the buffer gets reparsed after all of the commands output was produced, e.g like in the following code.

(defun compilation-reparse-buffer (&rest _ignored)
  (interactive)
  (compilation--flush-parse (point-min) (point-max))
  (compilation--ensure-parse (point-max)))

(add-hook 'weird-compilation-mode-hook
          (lambda nil
            (add-hook 'compilation-finish-functions #'compilation-reparse-buffer nil t)))