How to use font-lock to highlight end-of-line comments?

Question

I am trying to add a comment feature to json-mode. A comment starts with a hash sign and ends at the end of the line. However, if the hash sign is inside a string it should not count as the start of the comment. For example:

I am using Emacs 24.4, and installed json-mode from the package manager in Emacs.

Here is a minimal example:

$ emacs -Q -l init.el test.json

where init.el is:

(setq package-load-list '((json-mode t) (json-reformat t) (json-snatcher t)))
(require 'package)
(package-initialize)
(setq package-enable-at-startup nil)

(defun my-json-mode-comment-highlighter (limit)
  "When this function is called, it receives one argument `limit', the limit
of the search; it should begin searching at point, and not search beyond the limit.
It should return non-nil if it succeeds, and set the match data to describe
the match that was found. Returning nil indicates failure of the search.

Fontification will call function repeatedly with the same limit, 
and with point where the previous invocation left it, until function fails. 
On failure, this function need not reset point in any particular way."
  (let ((repeat 1)
        (noerror t))
    (re-search-forward "\\(#.*\\)$" limit noerror repeat)))

(defun my-json-mode-hook ()
  (let* ( ;; `sub-expression', specifies which subexpression of matcher should
         ;; be highlighted (instead of the entire text that matcher matched)
         (sub-expression 1)
         ;; If `override' is t, override existing fontification made by previous
         ;; elements of `font-lock-keywords'
         (override t)
         ;; If `mode' is nil, adds keywords to `font-lock-keywords' in the current buffer.
         (mode nil)
         (keywords
          `((my-json-mode-comment-highlighter
             ,sub-expression
             font-lock-comment-face
             ,override))))
    (font-lock-add-keywords mode keywords)))

(add-hook 'json-mode-hook #'my-json-mode-hook)

This will highlight comments correctly, but also incorrectly assume that a hash sign inside a string starts a comment. How can I extend the function my-json-mode-comment-highlighter so it works as desired?

Update:

I also tried to replace my-json-mode-hook with:

(defun my-json-mode-hook ()
  (modify-syntax-entry ?# "< b")
  (modify-syntax-entry ?\n "> b"))

But then nothing is highlighted.

Answer 1

(override t) is the culprit: that's what tells Emacs to apply the comment face to anything that starts with a # even if part of that thing has already been recognized as something else. Remove it.

However you shouldn't need to define a regex to highlight comments: Emacs understands comments out of the box, based on character syntax, as described by Lindydancer. It should be enough to declare # as a line comment start character with

(defun my-json-mode-hook ()
  (modify-syntax-entry ?# "< b"))

(and maybe (modify-syntax-entry ?\n "> b") if the JSON mode doesn't already define // as a line comment start.)

Answer 2

When using font-lock, things are colored in two phases:

• The syntactic phase highlights comments and string
• The keyword phase highlights anything else.

The syntactic phase is based on information in the syntax table. If things are not highlighted properly this mean that the syntax table is broken. There are plenty of examples of other languages like Ruby that use the same string and comment conventions a JSon.

Answer 3

Naive approach which will do slightly better: disallow quotes after # - #[^"']*$ instead of #.*$. This won't colorize comments containing quote but won't go inside string.

Proper approach: complicate the loop. Scan the line searching for any interesting special characters (those are ", ', \ and # unless I miss sth), keep track of whether you are inside some quotes, and find first # which is not inside quoted string.

Answer 4

I find that using the javascript (c++) comment style (/**/ and //) is more convenient and also much easier to add to json-mode. I was able to add support for this in json-mode.el with some very minor changes (the json-mode.el itself):

 (defconst json-font-lock-keywords-1
   (list
-   (list json-mode-quoted-key-re    1 font-lock-keyword-face)
+   (list json-mode-quoted-key-re    1 font-lock-keyword-face t)
-   (list json-mode-quoted-string-re 1 font-lock-string-face)
    (list json-mode-keyword-re       1 font-lock-constant-face)
    (list json-mode-number-re        1 font-lock-constant-face)
    )
   "Level one font lock.")

 ;;;###autoload                                                                                                                                                                                                                                                                                                                                                                                                  
 (define-derived-mode json-mode javascript-mode "JSON"
   "Major mode for editing JSON files"
-  (set (make-local-variable 'font-lock-defaults) '(json-font-lock-keywords-1 t)))
+  (setq font-lock-defaults '(json-font-lock-keywords-1)))

Explanation:

1. font-lock-defaults automatically becomes buffer-local when set, so you don't need that extra make-local-variable.
2. The 't' that I removed from the initialization of font-lock-defaults (the second element in the list) corresponds to the keywords-only in emacs docs. It basically means that if you DON'T set it then you inherit the string and comment fontification from the parent mode - javascript-mode, which has the exact comment syntax I want! :)
3. Now that we inherit the string fontification from js-mode, we no longer need to do it ourselves, so that's why I removed json-mode-quoted-string-re from json-font-lock-keywords-1.
4. The last important detail is that since we inherited all string syntax from javascript-mode, all the strings, including the keys will have the font-lock-string-face and not the font-lock-keyword-face. To remedy this, I used the override element when adding json-mode-quoted-key-re. If you look at the docs (where they talk about (matcher . subexp-highlighter)), you'll see that if override is t it simply replaces the previous fontification.

Answer 5

I agree with @Lindydancer that the best way would be to use the built-in syntax highlighting for highlighting comments. But in order to that, I think I would need to modify the source code of json-mode.el. Since I did not want to modify the source code of json-mode, I tried to do work around this using font-lock-add-keywords instead. It becomes a little bit complicated, but here is something that seems to work for me now:

(defun my-json-mode-comment-highlighter (limit)
  "When this function is called, it receives one argument `limit', the limit
of the search; it should begin searching at point, and not search beyond the limit.
It should return non-nil if it succeeds, and set the match data to describe
the match that was found. Returning nil indicates failure of the search.

Fontification will call function repeatedly with the same limit, 
and with point where the previous invocation left it, until function fails. 
On failure, this function need not reset point in any particular way."
  (let ((repeat 1)
        (noerror t)
        (string-pos-list nil)
        (found-match nil)
        (search-again t)
        (start-pos))
    (setq string-pos-list (my-json-mode--find-string-pos-list limit))
    (while search-again
      (if (re-search-forward "\\(#.*\\)$" limit noerror repeat)
          (progn
            (setq start-pos (match-beginning 1))
            (if (my-json-mode--check-outside-string string-pos-list start-pos)
                (progn
                  (setq found-match t)
                  (setq search-again nil))
              ;; comment was part of a string, so we move back from
              ;; the end of the line to the next character after the #
              (goto-char (+ start-pos 1)))) 
        (setq search-again nil))) ; re-search-forward failed.
    found-match))

(defun my-json-mode--check-outside-string (pos-list pos)
  "Checks if `pos' is outside any string on the current line."      
  (catch 'my-loop
    (dolist (elt pos-list)
      (when (<= (car elt) pos (cadr elt))
        (throw 'my-loop nil)))
    t))

(defun my-json-mode--find-string-pos-list (limit)
  "Find the starting and end positions of all strings in the current buffer
from (line-beginning-position) to `limit'. Exclude strings inside comments
  (comments start with a # sign). To avoid starting the search from
the beginning of the buffer, we assume that a string 
does not have newline characters."      
  (save-excursion
    (let ((noerror t)
          (repeat 1)
          (pos-list nil)
          (cur-pos nil))
      (beginning-of-line)
      (setq cur-pos (point))
      (while (re-search-forward json-mode-quoted-string-re limit noerror repeat)
        (let ((start-pos (match-beginning 1))
              (end-pos (match-end 1)))
          (if (my-json-mode--string-not-comment cur-pos start-pos)
              (push (list (match-beginning 1) (match-end 1)) pos-list)
            (end-of-line)) ; current line is a comment, so skip it
          (setq cur-pos end-pos)))
      pos-list)))

(defun my-json-mode--string-not-comment (start-pos end-pos)
  "Ensure that the string that starts at position `end-pos' is not part of a comment
starting after `start-pos' on the current line. We must have that `start-pos'
is strictly less than `end-pos'. If `start-pos' is not at the same line as
`end-pos', we set `start-pos' to (line-beginning-position)"   
  (let ((beg-line-pos (line-beginning-position))
        (noerror t)
        (repeat 1)
        (string-ok t))
    (when (< start-pos beg-line-pos)
      (setq start-pos beg-line-pos))
    (save-excursion
      (goto-char start-pos)
      (when (search-forward "#" end-pos noerror repeat)
        (setq string-ok nil)))
    string-ok))

(defun my-json-mode-hook ()
  (let* ( ;; `sub-expression', specifies which subexpression of matcher should
         ;; be highlighted (instead of the entire text that matcher matched)
         (sub-expression 1)
         ;; If `override' is t, override existing fontification made by previous
         ;; elements of `font-lock-keywords'
         (override nil)
         ;; If `mode' is nil, adds keywords to `font-lock-keywords' in the current buffer.
         (mode nil)
         (keywords
          `((my-json-mode-comment-highlighter
             ,sub-expression
             font-lock-comment-face
             ,override))))
    (font-lock-add-keywords mode keywords)))

(add-hook 'json-mode-hook #'my-json-mode-hook)