Why does inserting a newline break highlighting with syntax-propertize-function?

Question

I'm trying to write a major-mode that highlights triple-quoted strings. Here's a minimal reproducible example:

(defconst demo-triple-quoted-string-regex
  (rx "\"\"\""
      ;; After the delimiter, we're a sequence of
      ;; non-backslashes or blackslashes paired with something.
      (*? (or (not (any "\\"))
              (seq "\\" anything)))
      "\"\"\""))

(defun demo-stringify-triple-quote ()
  "Put `syntax-table' property on triple-quoted strings."
  (let* ((string-literal (match-string 0))
         (string-start-pos (- (point) (length string-literal)))
         (string-end-pos (point)))
    (unless (nth 4 (syntax-ppss)) ;; not inside comment
      (put-text-property string-start-pos string-end-pos
                         'syntax-table (string-to-syntax "|")))))

(defconst demo-syntax-propertize-function
  (syntax-propertize-rules
   (demo-triple-quoted-string-regex
    (0 (ignore (demo-stringify-triple-quote))))))

(define-derived-mode demo-mode prog-mode "Demo"
  "Major mode showing stack overflow question."
  (set (make-local-variable 'font-lock-defaults) '(()))
  (set (make-local-variable 'syntax-propertize-function)
       demo-syntax-propertize-function))

However, this leads to really bizarre behaviour when modifying the buffer. Here's my buffer contents:

dodgy when we put a newline after babel

"""
a
"
babel

"""

x = 1

M-x demo-mode gives correct highlighting:

but pressing enter suddenly gives this:

What am I doing wrong?

Answer 1

Thanks to Politza, and stepping through python-syntax-stringify with edebug, I've got this working. The changes were:

• | should only be applied to the first and last character in the triple-quoted-string.
• (Because syntax parsing is incremental?) It's not possible to search for a whole string. Instead, search for a delimiter and see whether or not you're in a triple-quoted string.

Working code:

(defconst demo-triple-quoted-string-regex
  (rx "\"\"\""))

(defun demo-stringify-triple-quote ()
  "Put `syntax-table' property on triple-quoted strings."
  (let* ((string-end-pos (point))
         (string-start-pos (- string-end-pos 3))
         (ppss (prog2
                   (backward-char 3)
                   (syntax-ppss)
                 (forward-char 3))))
    (unless (nth 4 (syntax-ppss)) ;; not inside comment
      (if (nth 8 (syntax-ppss))
          ;; We're in a string, so this must be the closing triple-quote.
          ;; Put | on the last " character.
          (put-text-property (1- string-end-pos) string-end-pos
                             'syntax-table (string-to-syntax "|"))
        ;; We're not in a string, so this is the opening triple-quote.
        ;; Put | on the first " character.
        (put-text-property string-start-pos (1+ string-start-pos)
                           'syntax-table (string-to-syntax "|"))))))

(defconst demo-syntax-propertize-function
  (syntax-propertize-rules
   (demo-triple-quoted-string-regex
    (0 (ignore (demo-stringify-triple-quote))))))

(define-derived-mode demo-mode prog-mode "Demo"
  "Major mode showing stack overflow question."
  (set (make-local-variable 'font-lock-defaults) '(()))
  (set (make-local-variable 'syntax-propertize-function)
       demo-syntax-propertize-function))