Why do regular expressions created with the regex builder use syntax different from the interactive regular expressions?

Question

So, using the regular expression builder (M-x re-builder), finding lines that end in \ takes "\\$", while in search and replace by regex, it only takes "\$". I would have expected the regex builder to build directly usable expressions, so what accounts for this difference?

Answer 1

There are actually four different re-builder syntax options, and you can switch between them with C-cTAB

Two are for the sexp-form regexp compilers rx and sregex (but as the former is more comprehensive and almost entirely syntax-compatible, you can really ignore sregex unless you happen to be working with old code that used it).

The other two syntax options are read (the default) and string (which is the syntax you use interactively).

The read syntax is the 'code' syntax -- i.e. as recognised by the lisp reader -- in which you enter the regexp as per the read syntax for strings:

C-hig (elisp) Syntax for Strings RET

The string syntax (which I've always considered an unnecessarily confusing name in this context) is the syntax of a regular expression string which has already been read, and which therefore does not have any of the character escaping required when writing the string. Which is to say, this is the actual regular expression syntax, the same as you use when Emacs prompts you interactively.

If you would like to use string syntax by default, add the following to your init file, or use M-x customize-option RET reb-re-syntax RET

(setq reb-re-syntax 'string)

Note that you can switch back and forth between read and string syntax when editing the regexp, with no loss of data. You can also switch from the sexp forms to read/string syntax (naturally; compiling sexps to strings is what those libraries are for), but you can't go in the other direction and generate a sexp from a string. re-builder remembers what the sexp was, so you don't lose that form when you change syntax; but it also doesn't get updated if you modify the regexp in a different syntax and then change back. In short, if you're building the regexp as a sexp, make sure you only edit it while using that syntax.

---

A gotcha with the rx support is that it's actually using the rx-to-string function, which isn't quite identical to using the rx macro in code. rx accepts an arbitrary number of form arguments and treats them as an implied sequence, whereas rx-to-string accepts only a single form, and any top-level sequence must be made explicit with '(sequence ...) or equivalent.

In short, when you enter a form '(...) in re-builder, it is processed as (rx-to-string '(...)) and not (rx ...)

Also take note that an invalid form might cause re-builder to stop dynamically updating the matches in the associated buffer, even after the form is made valid again. The C-cC-u binding for reb-force-update is useful for resolving these situations.

---

By default the mode line shows "RE Builder" when using read or string syntax, and "RE Builder Lisp" when using rx or sregex syntax, but it seems far more useful to identify the specific syntax in use (especially to differentiate between read and string).

If you install the delight package from GNU ELPA, you can use the following to add a syntax indicator to the mode line.

(let ((name '("Regexp[" (:eval (symbol-name reb-re-syntax)) "]")))
  (delight `((reb-mode ,name :major)
             (reb-lisp-mode ,name :major))))

This changes the mode name to "Regexp[read]" in read syntax, and likewise for the others.

Or to include a hint for the rx vs rx-to-string gotcha described above, make the mode line say "Regexp[rx-to-string]" when using the rx syntax:

(let ((name '("Regexp["
              (:eval (symbol-name (if (eq reb-re-syntax 'rx)
                                      'rx-to-string
                                    reb-re-syntax)))
              "]")))
  (delight `((reb-mode ,name :major)
             (reb-lisp-mode ,name :major))))