font-lock keywords and syntax-table mutually interfering

Question

I would like to define something that font-locks a bit like a new sort of string-literal/comment. I might write:

(font-lock-add-keywords 'emacs-lisp-mode
   (list '("<<.*>>" 0 'my-custom-face prepend)))

I'm using 'emacs-lisp-mode as my example just because it's an existing mode with syntax-table driven string literal and comment highlighting. As per How to highlight specific keywords inside strings/quotes? the prepend above ensures that my new delimiters take precedence, so that

<<"foo" bar>>

font locks entirely with the new face. I'm happy enough with that. But I would like

; some comment text <<foo>>

to highlight with the comment colors, and for

"some string stuff <<foo>>"

to highlight entirely with string-literal colors. Unfortunately, I get the custom face appearing within the literals and comments, which is distracting and arguably wrong (certainly, string literals and comments are mutually compatible). How do I get the outermost delimiters to take precedence?

(Annoyingly, if I don't have the prepend, the syntax-table stuff takes precedence particularly dramatically: something like <<foo bar>> is fine, but <<"foo" bar>> has the "foo" in string literal colors, and doesn't color the delimiters or the bar at all.)

Answer 1

I don't particularly like the solution, but I used a function matcher that called re-search-forward to find the start delimiter, and then checked the return code of syntax-ppss to see if the position was in a comment or string literal.

This looped forward until it ran out of room (i.e., respecting the provided limit parameter), or found a left-delimiter not inside comment or string syntax. Having done that, it could then search forward for the right delimiter and then return the appropriate match-data if it found one.