font-lock-keywords weirdness

Question

I am trying to modify pandoc-template-mode.el to deal with $$ which is an escape sequence for literal $ inside the Pandoc template. I have added the "\\$\\$" at the top but I still get bad font highlighting as if it were hitting last case of the function.

The modified pandoc-template-mode.el:

(defvar pandoc-template-font-lock-keywords
  '(("\\$\\$"
      (0 font-lock-keyword-face))
    ("\\$--.*"
      (0 font-lock-comment-face))
    ("\\(\\$\\)\\(if\\|for\\)(\\([^)]+\\))\\(\\$\\)"
      (1 font-lock-preprocessor-face)
      (2 font-lock-keyword-face)
      (3 font-lock-variable-name-face)
      (4 font-lock-preprocessor-face))
    ("\\(\\$\\)\\(endif\\|endfor\\|else\\)\\(\\$\\)"
      (1 font-lock-preprocessor-face)
      (2 font-lock-keyword-face)
      (3 font-lock-preprocessor-face))
    ("\\(\\$\\)\\(sep\\)\\(\\$\\)"
      (1 font-lock-preprocessor-face)
      (2 font-lock-builtin-face)
      (3 font-lock-preprocessor-face))
    ("\\(\\$\\)\\([^$]+\\)\\(\\$\\)"
      (1 font-lock-preprocessor-face)
      (2 font-lock-variable-name-face)
      (3 font-lock-preprocessor-face))
     )
  "Keyword highlighting specification for `pandoc-template-mode'.")

 ;;;###autoload
(define-derived-mode pandoc-template-mode fundamental-mode "Pandoc-Template"
  "A major mode for editing Pandoc-Template files."
  :syntax-table text-mode-syntax-table
  (setq-local font-lock-defaults
              '(pandoc-template-font-lock-keywords))
  (setq-local comment-start "$--")
  (setq-local comment-start-skip "\\$--[ \t]*")
  (setq-local comment-end "")
  (setq-local comment-end-skip "[ \t]*$"))

(provide 'pandoc-template-mode)

The test file with bad highlighting:

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<?aid style="50" type="snippet" readerVersion="6.0" featureSet="513" product="8.0(370)" ?>
<?aid SnippetType="InCopyInterchange"?>
<Document DOMVersion="8.0" Self="pandoc_doc">
    <RootCharacterStyleGroup Self="pandoc_character_styles">
      <CharacterStyle Self="$$ID/NormalCharacterStyle" Name="Default" />
      $charStyles$
    </RootCharacterStyleGroup>
    <RootParagraphStyleGroup Self="pandoc_paragraph_styles">
      <ParagraphStyle Self="$$ID/NormalParagraphStyle" Name="$$ID/NormalParagraphStyle"
          SpaceBefore="6" SpaceAfter="6"> <!-- paragraph spacing -->
        <Properties>
          <TabList type="list">
            <ListItem type="record">
              <Alignment type="enumeration">LeftAlign</Alignment>
              <AlignmentCharacter type="string">.</AlignmentCharacter>
              <Leader type="string"></Leader>
              <Position type="unit">10</Position> <!-- first tab stop -->
            </ListItem>
          </TabList>
        </Properties>
      </ParagraphStyle>
      $parStyles$
    </RootParagraphStyleGroup>
    <RootTableStyleGroup Self="pandoc_table_styles">
      <TableStyle Self="TableStyle/Table" Name="Table" />
    </RootTableStyleGroup>
    <RootCellStyleGroup Self="pandoc_cell_styles">
      <CellStyle Self="CellStyle/Cell" AppliedParagraphStyle="ParagraphStyle/$$ID/[No paragraph style]" Name="Cell" />
    </RootCellStyleGroup>
  <Story Self="pandoc_story"
      TrackChanges="false"
      StoryTitle="$if(title-prefix)$$title-prefix$ – $endif$$pagetitle$"
      AppliedTOCStyle="n"
      AppliedNamedGrid="n" >
    <StoryPreference OpticalMarginAlignment="true" OpticalMarginSize="12" />

<!-- body needs to be non-indented, otherwise code blocks are indented too far -->
$body$

  </Story>
  $hyperlinks$
</Document>

The issue is that on the 6th line of the template above (<CharacterStyle Self="$$ID/NormalCharacterStyle" Name="Default" />) the $$ is matched and colored but then the rest of the text should be uncolored. What actually happens is that the rest of the text after $$ is colored as if the last rule of the function matched as well.

Answer 1

Indeed, the last font-lock rule still matches, and of its 3 highlighting instructions, the first (that which highlights the opening $) won't be applied because this chars has already been highlighted by the first rule, but the other 2 highlighting instructions are still applied because that part of the text hasn't been highlighted yet.

There are also "tricky" cases such as:

$sep$$sep$

or

$$sep$sep$sep$$

where the $$ isn't "an escaped $". Maybe the best way to deal with it is to consider $...$ as a kind of "string":

(defvar pandoc-template-mode-syntax-table
  (let ((st (make-syntax-table)))
    (modify-syntax-entry ?$ "\"$")
    st))

and then in your rules you'll want to use things like:

 ("\\(\\$\\)\\(sep\\)\\(\\$\\)"
  (2 (if (save-excursion (nth 3 (syntax-ppss (match-beginngin 2))))
         font-lock-builtin-face)
     prepend))