3

I am trying to add a comment feature to json-mode. A comment starts with a hash sign and ends at the end of the line. However, if the hash sign is inside a string it should not count as the start of the comment. For example:

enter image description here

I am using Emacs 24.4, and installed json-mode from the package manager in Emacs.

Here is a minimal example:

$ emacs -Q -l init.el test.json

where init.el is:

(setq package-load-list '((json-mode t) (json-reformat t) (json-snatcher t)))
(require 'package)
(package-initialize)
(setq package-enable-at-startup nil)

(defun my-json-mode-comment-highlighter (limit)
  "When this function is called, it receives one argument `limit', the limit
of the search; it should begin searching at point, and not search beyond the limit.
It should return non-nil if it succeeds, and set the match data to describe
the match that was found. Returning nil indicates failure of the search.

Fontification will call function repeatedly with the same limit, 
and with point where the previous invocation left it, until function fails. 
On failure, this function need not reset point in any particular way."
  (let ((repeat 1)
        (noerror t))
    (re-search-forward "\\(#.*\\)$" limit noerror repeat)))

(defun my-json-mode-hook ()
  (let* ( ;; `sub-expression', specifies which subexpression of matcher should
         ;; be highlighted (instead of the entire text that matcher matched)
         (sub-expression 1)
         ;; If `override' is t, override existing fontification made by previous
         ;; elements of `font-lock-keywords'
         (override t)
         ;; If `mode' is nil, adds keywords to `font-lock-keywords' in the current buffer.
         (mode nil)
         (keywords
          `((my-json-mode-comment-highlighter
             ,sub-expression
             font-lock-comment-face
             ,override))))
    (font-lock-add-keywords mode keywords)))

(add-hook 'json-mode-hook #'my-json-mode-hook)

This will highlight comments correctly, but also incorrectly assume that a hash sign inside a string starts a comment. How can I extend the function my-json-mode-comment-highlighter so it works as desired?

Update:

I also tried to replace my-json-mode-hook with:

(defun my-json-mode-hook ()
  (modify-syntax-entry ?# "< b")
  (modify-syntax-entry ?\n "> b"))

But then nothing is highlighted.

  • 1
    Sounds like syntactic highlighting (which takes care of comments and strings) does not work perfectly for whatever JSON-editing mode you are using. Consider reporting that as a bug to the maintainer. IOW, this should just work, I think. – Drew Mar 5 '15 at 15:28
  • 1
    @Drew Actually, I think comments are not allowed in regular JSON files. So, comments is something I have added to the format as a private extension (that can be read by a Perl script). – Håkon Hægland Mar 5 '15 at 18:51
  • What mode is the buffer in? Which json-mode package is that — there's only a json in the default package list? If I copy your init.el then buffers for .json files are still in Javascript mode as they are by default. If I use your modify-syntax-entry-based hook and add it to js-mode-hook, the #… comment is highlighted as expected. – Gilles 'SO- stop being evil' Mar 5 '15 at 22:21
  • @Gilles I installed json-mode from the Emacs package manager. The GitHub page of the mode is github.com/joshwnj/json-mode. – Håkon Hægland Mar 6 '15 at 6:37
  • That function is overkill. Start from scratch and follow Gilles' answer. – Malabarba Mar 6 '15 at 17:11
7

(override t) is the culprit: that's what tells Emacs to apply the comment face to anything that starts with a # even if part of that thing has already been recognized as something else. Remove it.

However you shouldn't need to define a regex to highlight comments: Emacs understands comments out of the box, based on character syntax, as described by Lindydancer. It should be enough to declare # as a line comment start character with

(defun my-json-mode-hook ()
  (modify-syntax-entry ?# "< b"))

(and maybe (modify-syntax-entry ?\n "> b") if the JSON mode doesn't already define // as a line comment start.)

5

When using font-lock, things are colored in two phases:

  • The syntactic phase highlights comments and string
  • The keyword phase highlights anything else.

The syntactic phase is based on information in the syntax table. If things are not highlighted properly this mean that the syntax table is broken. There are plenty of examples of other languages like Ruby that use the same string and comment conventions a JSon.

  • Thanks @Lindydancer for the information. I tried to modify the syntax table, see my updated answer. But it did not help. – Håkon Hægland Mar 5 '15 at 18:43
  • 2
    @HåkonHægland, it appears that the JSON mode specifies "keyword only", i.e. skip the syntactic phase. Try adding (set (make-local-variable 'font-lock-defaults) '(json-font-lock-keywords-1)) to your hook. Also, the mode tries to highlight strings using a font-lock keyword, I would recommend that you would remove it and let the syntactic phase take care of it all together. – Lindydancer Mar 5 '15 at 22:55
1

Naive approach which will do slightly better: disallow quotes after # - #[^"']*$ instead of #.*$. This won't colorize comments containing quote but won't go inside string.

Proper approach: complicate the loop. Scan the line searching for any interesting special characters (those are ", ', \ and # unless I miss sth), keep track of whether you are inside some quotes, and find first # which is not inside quoted string.

1

I find that using the javascript (c++) comment style (/**/ and //) is more convenient and also much easier to add to json-mode. I was able to add support for this in json-mode.el with some very minor changes (the json-mode.el itself):

 (defconst json-font-lock-keywords-1
   (list
-   (list json-mode-quoted-key-re    1 font-lock-keyword-face)
+   (list json-mode-quoted-key-re    1 font-lock-keyword-face t)
-   (list json-mode-quoted-string-re 1 font-lock-string-face)
    (list json-mode-keyword-re       1 font-lock-constant-face)
    (list json-mode-number-re        1 font-lock-constant-face)
    )
   "Level one font lock.")

 ;;;###autoload                                                                                                                                                                                                                                                                                                                                                                                                  
 (define-derived-mode json-mode javascript-mode "JSON"
   "Major mode for editing JSON files"
-  (set (make-local-variable 'font-lock-defaults) '(json-font-lock-keywords-1 t)))
+  (setq font-lock-defaults '(json-font-lock-keywords-1)))

Explanation:

  1. font-lock-defaults automatically becomes buffer-local when set, so you don't need that extra make-local-variable.
  2. The 't' that I removed from the initialization of font-lock-defaults (the second element in the list) corresponds to the keywords-only in emacs docs. It basically means that if you DON'T set it then you inherit the string and comment fontification from the parent mode - javascript-mode, which has the exact comment syntax I want! :)
  3. Now that we inherit the string fontification from js-mode, we no longer need to do it ourselves, so that's why I removed json-mode-quoted-string-re from json-font-lock-keywords-1.
  4. The last important detail is that since we inherited all string syntax from javascript-mode, all the strings, including the keys will have the font-lock-string-face and not the font-lock-keyword-face. To remedy this, I used the override element when adding json-mode-quoted-key-re. If you look at the docs (where they talk about (matcher . subexp-highlighter)), you'll see that if override is t it simply replaces the previous fontification.
0

I agree with @Lindydancer that the best way would be to use the built-in syntax highlighting for highlighting comments. But in order to that, I think I would need to modify the source code of json-mode.el. Since I did not want to modify the source code of json-mode, I tried to do work around this using font-lock-add-keywords instead. It becomes a little bit complicated, but here is something that seems to work for me now:

(defun my-json-mode-comment-highlighter (limit)
  "When this function is called, it receives one argument `limit', the limit
of the search; it should begin searching at point, and not search beyond the limit.
It should return non-nil if it succeeds, and set the match data to describe
the match that was found. Returning nil indicates failure of the search.

Fontification will call function repeatedly with the same limit, 
and with point where the previous invocation left it, until function fails. 
On failure, this function need not reset point in any particular way."
  (let ((repeat 1)
        (noerror t)
        (string-pos-list nil)
        (found-match nil)
        (search-again t)
        (start-pos))
    (setq string-pos-list (my-json-mode--find-string-pos-list limit))
    (while search-again
      (if (re-search-forward "\\(#.*\\)$" limit noerror repeat)
          (progn
            (setq start-pos (match-beginning 1))
            (if (my-json-mode--check-outside-string string-pos-list start-pos)
                (progn
                  (setq found-match t)
                  (setq search-again nil))
              ;; comment was part of a string, so we move back from
              ;; the end of the line to the next character after the #
              (goto-char (+ start-pos 1)))) 
        (setq search-again nil))) ; re-search-forward failed.
    found-match))

(defun my-json-mode--check-outside-string (pos-list pos)
  "Checks if `pos' is outside any string on the current line."      
  (catch 'my-loop
    (dolist (elt pos-list)
      (when (<= (car elt) pos (cadr elt))
        (throw 'my-loop nil)))
    t))

(defun my-json-mode--find-string-pos-list (limit)
  "Find the starting and end positions of all strings in the current buffer
from (line-beginning-position) to `limit'. Exclude strings inside comments
  (comments start with a # sign). To avoid starting the search from
the beginning of the buffer, we assume that a string 
does not have newline characters."      
  (save-excursion
    (let ((noerror t)
          (repeat 1)
          (pos-list nil)
          (cur-pos nil))
      (beginning-of-line)
      (setq cur-pos (point))
      (while (re-search-forward json-mode-quoted-string-re limit noerror repeat)
        (let ((start-pos (match-beginning 1))
              (end-pos (match-end 1)))
          (if (my-json-mode--string-not-comment cur-pos start-pos)
              (push (list (match-beginning 1) (match-end 1)) pos-list)
            (end-of-line)) ; current line is a comment, so skip it
          (setq cur-pos end-pos)))
      pos-list)))

(defun my-json-mode--string-not-comment (start-pos end-pos)
  "Ensure that the string that starts at position `end-pos' is not part of a comment
starting after `start-pos' on the current line. We must have that `start-pos'
is strictly less than `end-pos'. If `start-pos' is not at the same line as
`end-pos', we set `start-pos' to (line-beginning-position)"   
  (let ((beg-line-pos (line-beginning-position))
        (noerror t)
        (repeat 1)
        (string-ok t))
    (when (< start-pos beg-line-pos)
      (setq start-pos beg-line-pos))
    (save-excursion
      (goto-char start-pos)
      (when (search-forward "#" end-pos noerror repeat)
        (setq string-ok nil)))
    string-ok))

(defun my-json-mode-hook ()
  (let* ( ;; `sub-expression', specifies which subexpression of matcher should
         ;; be highlighted (instead of the entire text that matcher matched)
         (sub-expression 1)
         ;; If `override' is t, override existing fontification made by previous
         ;; elements of `font-lock-keywords'
         (override nil)
         ;; If `mode' is nil, adds keywords to `font-lock-keywords' in the current buffer.
         (mode nil)
         (keywords
          `((my-json-mode-comment-highlighter
             ,sub-expression
             font-lock-comment-face
             ,override))))
    (font-lock-add-keywords mode keywords)))

(add-hook 'json-mode-hook #'my-json-mode-hook)
  • 2
    Or you could simply use font-lock-remove-keywords to remove the rule that highlights strings and let the syntactic phase do the job. Another alternative would be to write a new json-mode from scratch, the one on Melpa isn't that well written -- it doesn't provide indention support and the font-lock support is severely broken. My guess that you put more effort into the code you posted here that it would take to do a full rewrite. – Lindydancer Mar 6 '15 at 21:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.