Function matcher with font-lock-keywords anchored highlighter

Question

I can highlight text like the following:

constant:'variable,variable'

with code like this which uses the anchored matcher form:

(font-lock-add-keywords
 nil
 '(("\\(constant\\):'" (1 font-lock-constant-face t)
    ("\\(variable\\)[,']" nil nil (1 font-lock-variable-name-face t))))
 'append)

Which works fine. The 'constant' and 'variable' parts are highlighted correctly.

However, it fails on text like this:

constant:'variable,variable' constant:'variable,variable' constant:''

The first 'constant' is correctly matched by the anchor, the first 2 'variable' are matched by the variable regexp, but then it rampages on to match the other 'variable's, so when the anchor regexp resumes it can't match the second 'constant.

I guess I need to make the 'variable' matcher give up after it matches a closing quote. So I tried this:

(defun my-variable-search (limit)
  (if (save-excursion
        (re-search-backward "variable'\'" 0 'go))
      nil
    (re-search-forward "\\(variable\\)[,']" limit 'go)))

(font-lock-add-keywords
 nil
 '(("\\(constant\\):'" (1 font-lock-constant-face t)
    (my-variable-search nil nil (1 font-lock-variable-name-face t))))
 'append)

But it doesn't work.

Am I completely off track or have I made a stupid mistake?

Answer 1

From this in the documentation of font-lock-keywords, it sounds like you could use the PRE-MATCH-FORM to return the limit of the search for "variable", like this:

(font-lock-add-keywords nil
  '(("\\(constant\\):'" (1 font-lock-constant-face t)
     ("\\(variable\\)[,']"
      (save-excursion (search-forward-regexp "\\(variable,\\)*variable'"))
      nil
      (1 font-lock-variable-name-face t))))
  'append)

It sounds like it is acceptable for PRE-MATCH-FORM to modify match data, so save-match-data is probably not necessary here.

Here is the relevant documentation quote:

MATCH-ANCHORED should be of the form:

 (MATCHER PRE-MATCH-FORM POST-MATCH-FORM MATCH-HIGHLIGHT ...)

where MATCHER is a regexp to search for or the function name to call to make
the search, as for MATCH-HIGHLIGHT above, but with one exception; see below.
PRE-MATCH-FORM and POST-MATCH-FORM are evaluated before the first, and after
the last, instance MATCH-ANCHORED's MATCHER is used.  Therefore they can be
used to initialize before, and cleanup after, MATCHER is used.  Typically,
PRE-MATCH-FORM is used to move to some position relative to the original
MATCHER, before starting with MATCH-ANCHORED's MATCHER.  POST-MATCH-FORM might
be used to move back, before resuming with MATCH-ANCHORED's parent's MATCHER.

For example, an element of the form highlights (if not already highlighted):

 ("\\<anchor\\>" (0 anchor-face) ("\\<item\\>" nil nil (0 item-face)))

 discrete occurrences of "anchor" in the value of `anchor-face', and subsequent
 discrete occurrences of "item" (on the same line) in the value of `item-face'.
 (Here PRE-MATCH-FORM and POST-MATCH-FORM are nil.  Therefore "item" is
 initially searched for starting from the end of the match of "anchor", and
 searching for subsequent instances of "anchor" resumes from where searching
 for "item" concluded.)

The above-mentioned exception is as follows.  The limit of the MATCHER search
defaults to the end of the line after PRE-MATCH-FORM is evaluated.
However, if PRE-MATCH-FORM returns a position greater than the position after
PRE-MATCH-FORM is evaluated, that position is used as the limit of the search.
It is generally a bad idea to return a position greater than the end of the
line, i.e., cause the MATCHER search to span lines.