I am trying to enhance SQL mode such that editing of files of SQL code becomes more convenient, especially as far as jumping around is concerned (e.g. things like sql-goto-end-of-string etc.). I have found syntax parsing to be very useful in that, but have stumbled across one curious thing:

SQL has the apostrophe as the string delimiter and the (as would seem to me) quirky idea of using, of all possibilities, a double apostrophe to denote a literal apostrophe within a string, i.e. the ' doubles as its own escape character when it occurs two times in a row, such that 'McDonald''s' in SQL means what would be written as e.g. "McDonald's" or 'McDonald\'s' in some other languages. It would seem to mean to me that any SQL parser, when encountering an apostrophe inside a string, must always read the next character before it can decide whether this is the end of the string or rather a literal apostrophe inside it.

As to Emacs in sql-mode, (nth 3 (parse-partial-sexp (point-min) pos)) in sql-mode returns the following when parsing over such a construct (^ denotes point):

         ^    --> 39
          ^   --> nil
           ^  --> 39

In other words, it treats the sequence as two separate, abutting strings, 'McDonald' and 's' (which of course do not exist in valid SQL syntax), presumably because ' is simply the string terminator in sql-mode-syntax-table ((char-to-string (char-syntax ?')) returns "\"" and is simply described as " which means: string by describe-syntax).

My question is whether Emacs is in principle capable of parsing this construct correctly, if, e.g., given a more complex syntax table entry for '.

I see it is capable of parsing two-character comment delimiters, such that / has a specific meaning when followed by *, and * is specific when inside a comment and followed by /. The provisions in syntax tables for such sequences, which AFAIU seem to be the syntax flags, seem to be specific to comments, however, and the exact problem also seems to be different (I cannot think of an example of a single "end comment" character that is escaped with itself in the languages that I am familiar with, at least), as far as I can tell (I have not looked into prefix characters, admittedly).

I would readily accept if this is simply a limitation of Emacs' syntax parsing (and would not think it too hard to come up with workarounds for my purposes), but my impression is that syntax parsing is the most efficient method, so it would be really cool to do it this way if it can be done.

(I do wonder whether using syntax-table properties would work, e.g. by assigning the punctuation syntax class to '' within strings, and if so, how substantial the overhead would be.)

I would greatly appreciate any pointers through the jungle! Thanks!

  • Good, clear question.
    – Drew
    Mar 25, 2020 at 16:14

2 Answers 2


The syntax tables themselves can't handle this right, but Emacs offers syntax-propertize to circumvent this kind of limitation by giving special syntax to specific occurrences of characters in buffers.

E.g. pascal-mode (where the same escaping is used in strings as the one you describe) has:

(defconst pascal--syntax-propertize
   ;; The syntax-table settings are too coarse and end up treating /* and (/
   ;; as comment starters.  Fix it here by removing the "2" from the syntax
   ;; of the second char of such sequences.
   ("/\\(\\*\\)" (1 ". 3b"))
   ("(\\(/\\)" (1 (prog1 ". 1c" (forward-char -1) nil)))
   ;; Pascal uses '' and "" rather than \' and \" to escape quotes.
   ("''\\|\"\"" (0 (if (save-excursion
                         (nth 3 (syntax-ppss (match-beginning 0))))
                       (string-to-syntax ".")
                     ;; In case of 3 or more quotes in a row, only advance
                     ;; one quote at a time.
                     (forward-char -1)
(define-derived-mode pascal-mode ...
  (setq-local syntax-propertize-function pascal--syntax-propertize)
  • Good answer. So wouldn't it be a good idea for SQL mode(s) to do this, by default?
    – Drew
    Mar 25, 2020 at 16:14
  • If that's indeed the SQL way to do it, then yes. Patch welcome.
    – Stefan
    Mar 25, 2020 at 16:21
  • Thanks, Stefan and Drew!!! I am already working on the patch and will post my suggestion for scrutiny when I dare.
    – fsavigny
    Mar 25, 2020 at 18:17

Thanks, Stefan, it seems to work like a charm! I.e. the parser state is exactly as expected, and e.g. forward-sexp now jumps over 'McDonald''s', which it did not do without the syntax propertizing.

I am unsure where parse-sexp-lookup-properties, which must obviously be t for syntax table properties to be actually used, is set. pascal.el does not set it, and works fine (i.e. jumps over the above string -- of course I can't type any other Pascal). Is it t by default?

As I found the docstring of syntax-propertize-rules somewhat arcane, and the appropriate actions somewhat tricky to understand (pascal.el has a very crisp explanation) I have exhaustively commented the minimal rule I extracted from pascal.el, just in case somebody as amateurish as me needs a transparent example. A short version for the experts comes first:

(defconst sql--syntax-propertize-escaped-apostrophe
       (nth 3 (syntax-ppss            
           (match-beginning 0)))) 
     (string-to-syntax ".") ; AFAIU, just "." should also work
       (forward-char -1)

(add-hook 'sql-mode-hook
      #'(lambda ()
          (setq-local syntax-propertize-function

I'm dubious as to whether the mode hook is the right place for a patch, as I've never submitted one before.

The long version:

(defconst sql--syntax-propertize-escaped-apostrophe
   ;; Out of many possible RULES, we need just one, which propertizes
   ;; escaped apostrophes within strings (such as the '' in
   ;; 'McDonald''s') with the appropriate syntax class (punctuation):

   ("''" ;; A rule's car is a REGEXP; here: when you find two ' in a row ...

    (0 ;; ... apply to "HIGHLIGHT", i.e. subexp
       ;; (here NUMBER 0, i.e., to the whole "''" match), ...

     ;; ... the following SYNTAX property, i.e. what the following
     ;; expression returns (should be the appropriate syntax-table
     ;; property when appropriate, or nil, when not appropriate):

     (save-excursion                  ; (Restore point after you have peeked to the left.)
       (nth 3 (syntax-ppss            ; Are we inside a string ...
           (match-beginning 0))))     ; ... at the pos directly before the ''?
     ;; then return the punctuation syntax-table property:
     (string-to-syntax ".") ; AFAIU, just "." should also work

       ;; Else, the two '' found are not an escaped ' within a string,
       ;; so we need to return nil, whatever else may need to be done.
       ;; What the strings *can be* is one of the following:
       ;; a) just '' on its own, i.e. an empty string, in which case
       ;;    we can just search on (apart from returning nil)
       ;; b) part of any number of '''' ... within a comment, in which
       ;;    case we can also just search on (apart from returning nil) 
       ;; c) the first two of four '''' (the whole sequence being a
       ;;    lonely escaped apostrophe in a string), in which case we
       ;;    must not propertize the first two, but the 2nd and 3rd
       ;;    apostrophes. Thus, we'll return nil for now, but need to
       ;;    go back one char, so the next search continues before the
       ;;    2nd ' and will then match the 2nd + 3rd '. (Otherwise, the
       ;;    1st + 2nd and then the 3rd + 4th will be examined
       ;;    together, missing exactly the sequence of the 2nd +
       ;;    3rd.)
       ;; We can actually go back one character in *all* of these cases,
       ;; however, because this causes no harm in the first two cases:
       ;; in case a), point is then followed by just one ', so won't
       ;; lead to any match on the next search, while in the esoteric
       ;; case b), point will go through a ''''... sequence of any
       ;; length at half the speed (i.e. twice the searches), but our
       ;; propertizing expression will return nil on each and every of
       ;; the overlapping '' sequences it then matches, as (nth 3
       ;; (syntax-ppss POS)) invariably returns nil when POS is within
       ;; a comment, unimpressed by any char that is normally a string
       ;; terminator. Distinguishing between case c) and the other two
       ;; would very likely cause more overhead than the simple
       ;; backtracking used here (especially as case b) is certainly
       ;; extremely rare, although one should never say it never
       ;; occurs).
       ;; Because of the backtracking we do, the '' *cannot* be:
       ;; - the 3rd and 4th ' of a '''' sequence, because because of
       ;;   our backtracking, a search can only start before the 1st,
       ;;   before the 2nd, and before the 4th '.
       ;; So, as explained, now go back one char and return nil:
       (forward-char -1)
  • Sorry about the hook. Of course it needs to go into the body of define-derived-mode.
    – fsavigny
    Mar 25, 2020 at 21:12
  • Some clarifications, in response to your commented code: using just "." wouldn't work (it works if it's the whole expression, but not if it's just the value returned by the expression); and I wouldn't call the (forward-char -1) backtracking: it just makes the failed match advance by 1 char instead of 2 since the "current match" was found at (match-beginning 0)` and (forward-char -1) makes us try again one char after (match-beginning 0)`.
    – Stefan
    Mar 26, 2020 at 4:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.