2

I am trying to use a regexp to match strings, e.g. a pattern that matches these kinds of quoted strings:

"test" or 'test' or 'test with "quote"' or "test with 'quote'"

This pattern matches all those,

"\\(\"\\|'\\)[^\\1]+?\\1"

but, it fails on "test\n" (and other control characters like \t and \r). I don't understand why \n (a newline) is so special it doesn't match [^\\1] which I thought was a character that doesn't match the opening quote. Is this expected?

If I replace [^\\1] with . then it works on all the strings, including the one with \n in it. I guess it is ok because +? makes it nongreedy so it seems to not over match.

Note, this question originated from How to highlight in different colors for variables inside `fstring` on python-mode.

13
  • Would you mind linking where I can see what ^\1 matches? regexps and their docs give me a migraine every time I need to use one.
    – RichieHH
    Feb 17 '20 at 14:07
  • 1
    Are (negated) backreferences allowed inside character alternatives in Emacs regular expressions? The manual neither confirms or denies that, as far as I can tell (gnu.org/software/emacs/manual/html_node/elisp/…)
    – rpluim
    Feb 17 '20 at 14:14
  • @RichieHH \\1 should match whatever is in the first group, which in this case is (\"\|'), i.e. a " or a '. The syntax [^\1] (I thought) would match any character that is not \1. Feb 17 '20 at 14:20
  • 1
    And one more bit of info: '\' is not special inside characacter alternatives according to the last paragraph of gnu.org/software/emacs/manual/html_node/emacs/Regexps.html , so I think that means backreferences aren't allowed.
    – rpluim
    Feb 17 '20 at 15:15
  • 1
    @Hubisan, your [^\\1] is matching 'anything except backslash or 1'.
    – rpluim
    Feb 17 '20 at 15:53
3

[^\\1] and [^\1] match the same thing: any single character except 1 or \. The doc is clear about this. \ and 1 are not special inside [...].

Negation, other than in a character alternative, is not possible using a regular expression. Use Lisp code instead. For example, find something that might include what you don't want, test it, and exclude it if it's something you don't want.

(Suggestion: describe your real problem - the problem for which you thought regexp-searching would provide a solution directly.)

3
  • Actually, [^\1] matches anything excepts Ctrl-A.
    – rpluim
    Feb 17 '20 at 16:10
  • @rpluim No, [^\1] and [\\1] do what Drew said if this character sequence is not part of a string to be read by the Elisp reader. Only the reader translates \1 within a string to the character with code 1. Drew did not say anything about that.
    – Tobias
    Feb 17 '20 at 17:00
  • @Tobias Oh, I was testing in ielm, that explains it.
    – rpluim
    Feb 17 '20 at 17:52
4

You asked about regexps. But from my perspective this is rather an X-Y-problem.

This is an answer to your real question how to find strings in buffers with major mode derived from prog-mode assuming that its syntax table is set-up.

Instead of trying to compose a suitable regexp you can use the syntax parser built-in to Emacs. The following code shows how you can do that.

The main tools are:

  • syntax-ppss for requesting the current state of the syntax parser
  • parse-partial-sexp for fleeing comments and for finding the beginning of strings (and comments)
  • scan-sexp for finding the end of the string

Note, that it should be possible to use this function as MATCHER for font-lock-keywords and related.

(defun my-find-next-string (&optional bound)
  "Find the next string up to BOUND.
BOUND defaults to `point-max'.
If we start within a string we skip to its end
and start there with the search."
  (unless bound
    (setq bound (point-max)))
  (let ((state (syntax-ppss))
    start
    end)
    ;; starting within string or comment:
    (when (nth 8 state)
      (setq state (parse-partial-sexp (point)
                      bound
                      nil nil
                      state
                      'syntax-table)))
    ;; searching for strings, skipping comments
    (while (and
        (setq state (parse-partial-sexp (point)
                        bound
                        nil nil
                        state
                        'syntax-table))
        (< (point) bound)
        (null (setq start (and (nth 3 state)
                   (nth 8 state)))))) ;; inside string
    (when start
      (unwind-protect
      (set-match-data
       (list (goto-char start)
         (goto-char (scan-sexps (point) 1))
         ))
    nil)
      (point))))
4
  • 1
    +1. But it depends on the current syntax defining "string" as indicated in the Q, i.e., double-quoted or single-quoted, with possibly embedded single-quoted or double-quoted, respectively. (But how many such nestings are allowed? Unspecified.) (And I agree with you about this likely being an X-Y problem.)
    – Drew
    Feb 17 '20 at 18:30
  • @Drew This is not really about nestings. In a double-quoted string you can use as many single-quotes you want but you need to escape every double-quote. In a single-quoted string you can use double-quotes without escape but you must escape single-quotes.
    – Tobias
    Feb 17 '20 at 20:34
  • One can imagine a syntax that allows, for example (and for whatever reason) a string with nested syntax, such as "aaa 'bb "ccc cc" bbb' aaaa". The question isn't specific about what "string" syntax to respect. That's what I meant about the syntax not being well specified - OP just gave some examples.
    – Drew
    Feb 17 '20 at 22:11
  • @Drew Okay fair enough. I referred in my comment to the Python context which the OP gave.
    – Tobias
    Feb 18 '20 at 3:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.