Does any package exist that can make interactive emacs-style regular expressions more readable by using font-locking?

Question

Ideally the package would allow one to display emacs-style regular expressions, as they are entered interactively during a search and replace command, in a more human-readable form.

Perhaps fewer backslash escape characters would be needed if color/bold/italics/underline could be leveraged optionally instead.

Making reg-exps easier to enter by allowing the user to choose some settings based on the type of text file one is editing, again to reduce the number of backslashes that are needed, would be desirable as well.

The rx package is nice, but a little too verbose for everyday use.

Answer 1

TLDR: If you want to minimize the backslashes, then your best bet is to use pcre2el: convert between PCRE, Emacs and rx regexp syntax, specifically the pcre-mode.

The screenshot below demonstrates that using pcre-mode indeed produces more readable regexp-s---the regexps are shorter; and have less backslash-noise.

Addtional Note:

The following long reply is tailored to people who know nothing about pcre, but have some understanding of Emacs-regexps. This is the reason why the screenshots start with an elisp-regexp, generates a pcre-regexp, and uses that subsequently.)

If someone in the audience know about pcre-style regexp, they can use the menu I present to move on from pcre to elisp-style regexps, and even hack *.el source code that uses regexp-s. (*.el source files have one more level of backslash, LOL)

See also Limitations on PCRE syntax

---

If you want to minimize the backslashes, then your best bet is to use pcre2el: convert between PCRE, Emacs and rx regexp syntax, specifically the pcre-mode.

1. M-x package-install RET pcre2el RET
2. Copy the below snippet to my-pcre2el-menu.el
3. Now do M-x load-file RET my-pcre2el-menu.el; you will see menu for pcre2el library. See below for screenshots.

(require 'pcre2el)

;;; Menu for Keymaps

;;;; Menu for Keymap `rxt--read-pcre-mode-map'

(easy-menu-define my-rxt--read-pcre-mode-map-menu rxt--read-pcre-mode-map "Menu for Rxt  Read Pcre Mode Map."
  '("My Rxt  Read Pcre Mode Map"
    ("Minor Mode P"
     [" Rxt  Toggle S Mode" rxt--toggle-s-mode :style toggle :selected rxt--toggle-s-mode :help "Toggle emulated PCRE single-line (s) flag."]
     [" Rxt  Toggle X Mode" rxt--toggle-x-mode :style toggle :selected rxt--toggle-x-mode :help "Toggle emulated PCRE extended (x) flag."]
     [" Rxt  Toggle I Mode" rxt--toggle-i-mode :style toggle :selected rxt--toggle-i-mode :help "Toggle emulated PCRE case-insensitive (i) flag."])))

;;;; Menu for Keymap `rxt-help-mode-map'

(easy-menu-define my-rxt-help-mode-map-menu rxt-help-mode-map "Menu for Rxt Help Mode Map."
  '("My Rxt Help Mode Map"
    ["  Down List" down-list :help "(down-list &optional ARG INTERACTIVE)\n\nMove forward down one level of parentheses.\nThis command will also work on other parentheses-like expressions\ndefined by the current language mode.\nWith ARG, do this that many times.\nA negative argument means move backward but still go down a level.\nThis command assumes point is not in a string or comment.\nIf INTERACTIVE is non-nil, as it is interactively,\nreport errors as appropriate for this kind of usage."]
     [" Backward Up List" backward-up-list :help "(backward-up-list &optional ARG ESCAPE-STRINGS NO-SYNTAX-CROSSING)\n\nMove backward out of one level of parentheses.\nThis command will also work on other parentheses-like expressions\ndefined by the current language mode.  With ARG, do this that\nmany times.  A negative argument means move forward but still to\na less deep spot.\n\nIf ESCAPE-STRINGS is non-nil (as it is interactively), move out\nof enclosing strings as well.\n\nIf NO-SYNTAX-CROSSING is non-nil (as it is interactively), prefer\nto break out of any enclosing string instead of moving to the\nstart of a list broken across multiple strings.\n\nOn error, location of point is unspecified."]
     [" Backward List" backward-list :help "(backward-list &optional ARG INTERACTIVE)\n\nMove backward across one balanced group of parentheses.\nThis command will also work on other parentheses-like expressions\ndefined by the current language mode.\nWith ARG, do it that many times.\nNegative arg -N means move forward across N groups of parentheses.\nThis command assumes point is not in a string or comment.\nIf INTERACTIVE is non-nil, as it is interactively,\nreport errors as appropriate for this kind of usage."]
     [" Forward List" forward-list :help "(forward-list &optional ARG INTERACTIVE)\n\nMove forward across one balanced group of parentheses.\nThis command will also work on other parentheses-like expressions\ndefined by the current language mode.\nWith ARG, do it that many times.\nNegative arg -N means move backward across N groups of parentheses.\nThis command assumes point is not in a string or comment.\nIf INTERACTIVE is non-nil, as it is interactively,\nreport errors as appropriate for this kind of usage."]
     [" Previous Line" previous-line :help "(previous-line &optional ARG TRY-VSCROLL)\n\nMove cursor vertically up ARG lines.\nInteractively, vscroll tall lines if `auto-window-vscroll' is enabled.\nNon-interactively, use TRY-VSCROLL to control whether to vscroll tall\nlines: if either `auto-window-vscroll' or TRY-VSCROLL is nil, this\nfunction will not vscroll.\n\nARG defaults to 1.\n\nIf there is no character in the target line exactly over the current column,\nthe cursor is positioned after the character in that line that spans this\ncolumn, or at the end of the line if it is not long enough.\n\nIf the variable `line-move-visual' is non-nil, this command moves\nby display lines.  Otherwise, it moves by buffer lines, without\ntaking variable-width characters or continued lines into account.\nSee \\[previous-logical-line] for a command that always moves by buffer lines.\n\nThe command \\[set-goal-column] can be used to create\na semipermanent goal column for this command.\nThen instead of trying to move exactly vertically (or as close as possible),\nthis command moves to the specified goal column (or as close as possible).\nThe goal column is stored in the variable `goal-column', which is nil\nwhen there is no goal column.  Note that setting `goal-column'\noverrides `line-move-visual' and causes this command to move by buffer\nlines rather than by display lines."]
     [" Next Line" next-line :help "(next-line &optional ARG TRY-VSCROLL)\n\nMove cursor vertically down ARG lines.\nInteractively, vscroll tall lines if `auto-window-vscroll' is enabled.\nNon-interactively, use TRY-VSCROLL to control whether to vscroll tall\nlines: if either `auto-window-vscroll' or TRY-VSCROLL is nil, this\nfunction will not vscroll.\n\nARG defaults to 1.\n\nIf there is no character in the target line exactly under the current column,\nthe cursor is positioned after the character in that line that spans this\ncolumn, or at the end of the line if it is not long enough.\nIf there is no line in the buffer after this one, behavior depends on the\nvalue of `next-line-add-newlines'.  If non-nil, it inserts a newline character\nto create a line, and moves the cursor to that line.  Otherwise it moves the\ncursor to the end of the buffer.\n\nIf the variable `line-move-visual' is non-nil, this command moves\nby display lines.  Otherwise, it moves by buffer lines, without\ntaking variable-width characters or continued lines into account.\nSee \\[next-logical-line] for a command that always moves by buffer lines.\n\nThe command \\[set-goal-column] can be used to create\na semipermanent goal column for this command.\nThen instead of trying to move exactly vertically (or as close as possible),\nthis command moves to the specified goal column (or as close as possible).\nThe goal column is stored in the variable `goal-column', which is nil\nwhen there is no goal column.  Note that setting `goal-column'\noverrides `line-move-visual' and causes this command to move by buffer\nlines rather than by display lines."]
     [" Kill This Buffer" kill-this-buffer :help "Kill the current buffer.\nWhen called in the minibuffer, get out of the minibuffer\nusing `abort-recursive-edit'.\n\nThis command can be reliably invoked only from the menu bar,\notherwise it could decide to silently do nothing."]
     [" Quit Window" quit-window :help "(quit-window &optional KILL WINDOW)\n\nQuit WINDOW and bury its buffer.\nWINDOW must be a live window and defaults to the selected one.\nWith prefix argument KILL non-nil, kill the buffer instead of\nburying it.\n\nThis calls the function `quit-restore-window' to delete WINDOW or\nshow some other buffer in it.  See Info node `(elisp) Quitting\nWindows' for more details.\n\nThe functions in `quit-window-hook' will be run before doing\nanything else."]))

;;;; Menu for Keymap `rxt-mode-map'

(easy-menu-define my-rxt-mode-map-menu rxt-mode-map "Menu for Rxt Mode Map."
  '("Rxt Mode Map"
    ["  Pcre Query Replace Regexp" pcre-query-replace-regexp :help "Perform `query-replace-regexp' using PCRE syntax.\n\nConsider using `pcre-mode' instead of this function."]
    "----------------"
    ("Rxt Elisp")
    ["  Rxt Elisp To Pcre" rxt-elisp-to-pcre :help "(rxt-elisp-to-pcre REGEXP)\n\nTranslate REGEXP, a regexp in Emacs Lisp syntax, to Perl-compatible syntax.\n\nInteractively, reads the regexp in one of three ways. With a\nprefix arg, reads from minibuffer without string escaping, like\n`query-replace-regexp'. Without a prefix arg, uses the text of\nthe region if it is active. Otherwise, uses the result of\nevaluating the sexp before point (which might be a string regexp\nliteral or an expression that produces a string).\n\nDisplays the translated PCRE regexp in the echo area and copies\nit to the kill ring.\n\nEmacs regexp features such as syntax classes which cannot be\ntranslated to PCRE will cause an error."]
    ["  Rxt Elisp To Rx" rxt-elisp-to-rx :help "(rxt-elisp-to-rx REGEXP)\n\nTranslate REGEXP, a regexp in Emacs Lisp syntax, to `rx' syntax.\n\nSee `rxt-elisp-to-pcre' for a description of the interactive\nbehavior and `rx' for documentation of the S-expression based\nregexp syntax."]
    ["  Rxt Elisp To Strings" rxt-elisp-to-strings :help "(rxt-elisp-to-strings REGEXP)\n\nReturn a list of all strings matched by REGEXP, an Emacs Lisp regexp.\n\nSee `rxt-elisp-to-pcre' for a description of the interactive behavior.\n\nThis is useful primarily for getting back the original list of\nstrings from a regexp generated by `regexp-opt', but it will work\nwith any regexp without unbounded quantifiers (*, +, {2, } and so\non).\n\nThrows an error if REGEXP contains any infinite quantifiers."]
    "----------------"
    ["  Rxt Toggle Elisp Rx" rxt-toggle-elisp-rx :help "Toggle the regexp near point between Elisp string and rx syntax."]
    "----------------"
    ("Rxt Pcre")
    ["  Rxt Pcre To Elisp" rxt-pcre-to-elisp :help "(rxt-pcre-to-elisp PCRE &optional FLAGS)\n\nTranslate PCRE, a regexp in Perl-compatible syntax, to Emacs Lisp.\n\nInteractively, uses the contents of the region if it is active,\notherwise reads from the minibuffer. Prints the Emacs translation\nin the echo area and copies it to the kill ring.\n\nPCRE regexp features that cannot be translated into Emacs syntax\nwill cause an error. See the commentary section of pcre2el.el for\nmore details."]
    ["  Rxt Pcre To Rx" rxt-pcre-to-rx :help "(rxt-pcre-to-rx PCRE &optional FLAGS)\n\nTranslate PCRE, a regexp in Perl-compatible syntax, to `rx' syntax.\n\nSee `rxt-pcre-to-elisp' for a description of the interactive behavior."]
    ["  Rxt Pcre To Strings" rxt-pcre-to-strings :help "(rxt-pcre-to-strings PCRE &optional FLAGS)\n\nReturn a list of all strings matched by PCRE, a Perl-compatible regexp.\n\nSee `rxt-elisp-to-pcre' for a description of the interactive\nbehavior and `rxt-elisp-to-strings' for why this might be useful.\n\nThrows an error if PCRE contains any infinite quantifiers."]
    "----------------"
    ("Rxt Explain")
    ["  Rxt Explain" rxt-explain :help "Pop up a buffer with pretty-printed `rx' syntax for the regex at point.\n\nChooses regex syntax to read based on current major mode, calling\n`rxt-explain-elisp' if buffer is in `emacs-lisp-mode' or\n`lisp-interaction-mode', or `rxt-explain-pcre' otherwise."]
    ["  Rxt Explain Pcre" rxt-explain-pcre :help "(rxt-explain-pcre REGEXP &optional FLAGS)\n\nInsert the pretty-printed `rx' syntax for REGEXP in a new buffer.\n\nREGEXP is a regular expression in PCRE syntax. See\n`rxt-pcre-to-elisp' for a description of how REGEXP is read\ninteractively."]
    ["  Rxt Explain Elisp" rxt-explain-elisp :help "(rxt-explain-elisp REGEXP)\n\nInsert the pretty-printed `rx' syntax for REGEXP in a new buffer.\n\nREGEXP is a regular expression in Emacs Lisp syntax. See\n`rxt-elisp-to-pcre' for a description of how REGEXP is read\ninteractively."]
    "----------------"
    ("Rxt Convert")
    ["  Rxt Convert Syntax" rxt-convert-syntax :help "Convert regex at point to other kind of syntax, depending on major mode.\n\nFor buffers in `emacs-lisp-mode' or `lisp-interaction-mode',\ncalls `rxt-elisp-to-pcre' to convert to PCRE syntax. Otherwise,\ncalls `rxt-pcre-to-elisp' to convert to Emacs syntax.\n\nThe converted syntax is displayed in the echo area and copied to\nthe kill ring; see the two functions named above for details."]
    ["  Rxt Convert To Rx" rxt-convert-to-rx :help "Convert regex at point to RX syntax. Chooses Emacs or PCRE syntax by major mode."]
    ["  Rxt Convert To Strings" rxt-convert-to-strings :help "Convert regex at point to RX syntax. Chooses Emacs or PCRE syntax by major mode."]))

;;; Menu for Libraries

;;; Menu for Package `("pcre2el" "pcre2el-tests")'

(easy-menu-define my-pcre2el-menu global-map "Menu for Pcre2el."
  '("My Pcre2el"
    ["Rxt Help Mode" rxt-help-mode :help "Major mode derived from `emacs-lisp-mode' by `define-derived-mode'.\nIt inherits all of the parent's attributes, but has its own keymap,\nabbrev table and syntax table:\n\n  `rxt-help-mode-map', `rxt-help-mode-abbrev-table' and\n`rxt-help-mode-syntax-table'\n\nwhich more-or-less shadow emacs-lisp-mode's corresponding tables.\n\nIn addition to any hooks its parent mode might have run, this mode\nruns the hook `rxt-help-mode-hook', as the final or penultimate step\nduring initialization.\n\n\\{rxt-help-mode-map}"]
    "----------------"
    ("Minor Mode P")
    ["  Turn On Rxt Mode" turn-on-rxt-mode :style toggle :selected turn-on-rxt-mode :help "Turn on `rxt-mode' in the current buffer."]
    ["  Rxt Mode" rxt-mode :style toggle :selected rxt-mode :help "(rxt-mode &optional ARG)\n\nRegex translation utilities.\n\nThis is a minor mode.  If called interactively, toggle the `Rxt\nmode' mode.  If the prefix argument is positive, enable the mode,\nand if it is zero or negative, disable the mode.\n\nIf called from Lisp, toggle the mode if ARG is `toggle'.  Enable\nthe mode if ARG is nil, omitted, or is a positive number.\nDisable the mode if ARG is a negative number.\n\nTo check whether the minor mode is enabled in the current buffer,\nevaluate `rxt-mode'.\n\nThe mode's hook is called both when the mode is enabled and when\nit is disabled."]
    ["  Rxt Global Mode" rxt-global-mode :style toggle :selected rxt-global-mode :help "(rxt-global-mode &optional ARG)\n\nToggle Rxt mode in all buffers.\nWith prefix ARG, enable Rxt-Global mode if ARG is positive; otherwise,\ndisable it.\n\nIf called from Lisp, toggle the mode if ARG is `toggle'.\nEnable the mode if ARG is nil, omitted, or is a positive number.\nDisable the mode if ARG is a negative number.\n\nRxt mode is enabled in all buffers where `turn-on-rxt-mode' would do\nit.\n\nSee `rxt-mode' for more information on Rxt mode."]
    ["  Rxt  Read Pcre Mode" rxt--read-pcre-mode :style toggle :selected rxt--read-pcre-mode :help "(rxt--read-pcre-mode &optional ARG)\n\nMinor-mode with key-bindings for toggling PCRE flags.\n\nYou should not normally call this directly.  It will be enabled\nin minibuffers for `read-regexp' and in the `re-builder' buffer\nwhen `pcre-mode' is active.  These bindings will also be added to\n`isearch-mode-map' in `pcre-mode'.\n\nThis is a minor mode.  If called interactively, toggle the\n`Rxt--Read-Pcre mode' mode.  If the prefix argument is positive,\nenable the mode, and if it is zero or negative, disable the mode.\n\nIf called from Lisp, toggle the mode if ARG is `toggle'.  Enable\nthe mode if ARG is nil, omitted, or is a positive number.\nDisable the mode if ARG is a negative number.\n\nTo check whether the minor mode is enabled in the current buffer,\nevaluate `rxt--read-pcre-mode'.\n\nThe mode's hook is called both when the mode is enabled and when\nit is disabled."]
    ["  Pcre Mode" pcre-mode :style toggle :selected pcre-mode :help "(pcre-mode &optional ARG)\n\nUse emulated PCRE syntax for regexps wherever possible.\n\nAdvises the `interactive' specs of `read-regexp' and the\nfollowing other functions so that they read PCRE syntax and\ntranslate to its Emacs equivalent:\n\n- `align-regexp'\n- `find-tag-regexp'\n- `sort-regexp-fields'\n- `isearch-message-prefix'\n- `ibuffer-do-replace-regexp'\n\nAlso alters the behavior of `isearch-mode' when searching by regexp.\n\nThis is a global minor mode.  If called interactively, toggle the\n`PCRE mode' mode.  If the prefix argument is positive, enable the\nmode, and if it is zero or negative, disable the mode.\n\nIf called from Lisp, toggle the mode if ARG is `toggle'.  Enable\nthe mode if ARG is nil, omitted, or is a positive number.\nDisable the mode if ARG is a negative number.\n\nTo check whether the minor mode is enabled in the current buffer,\nevaluate `(default-value \\='pcre-mode)'.\n\nThe mode's hook is called both when the mode is enabled and when\nit is disabled."]
    "----------------"
    ("Pcre2el")
    ["  Pcre To Elisp" pcre-to-elisp :help "(pcre-to-elisp PCRE &optional FLAGS)\n\nTranslate PCRE, a regexp in Perl-compatible syntax, to Emacs Lisp.\n\nInteractively, uses the contents of the region if it is active,\notherwise reads from the minibuffer. Prints the Emacs translation\nin the echo area and copies it to the kill ring.\n\nPCRE regexp features that cannot be translated into Emacs syntax\nwill cause an error. See the commentary section of pcre2el.el for\nmore details."]
    ))

(provide 'my-pcre2el-menu)

3. Copy the following content to to *scratch* buffer. (org-mode is notorious for its use of regexps. For the sake of illustration I chose the regexp for parsing American-style regexp , and added some American date-s for test driving the pcre2el library)

;; American style dates for matching the regexp
5/30

5/30/7

"^ *\\(0?[1-9]\\|1[012]\\)/\\(0?[1-9]\\|[12][0-9]\\|3[01]\\)\\(/\\([0-9]+\\)\\)?\\([^/0-9]\\|$\\)"

What you see below is the output of xr on the above regexp. (See xr: A new weapon for your regex arsenal)

(seq bol
     (zero-or-more " ")
     (group
      (or
       (seq
    (opt "0")
    (any "1-9"))
       (seq "1"
        (any "012"))))
     "/"
     (group
      (or
       (seq
    (opt "0")
    (any "1-9"))
       (seq
    (any "12")
    (any "0-9"))
       (seq "3"
        (any "01"))))
     (opt
      (group "/"
         (group
          (one-or-more
           (any "0-9")))))
     (group
      (or
       (not
    (any "0-9" "/"))
       eol)))

4. When in *scratch-buffer*, M-x load-file RET my-pcre2el-menu.el.

5. Enable pcre-mode

6. Enable rxt-mode

7. Put the cursor on the regexp string, and do rxt-elisp-to-pcre

You will see the following in the *Messages* buffer. Make a note of the pcre version of the above elisp regexp. We will use it later.

PCRE-version of above elisp regexp

^ *(0?[1-9]|1[210])/(0?[1-9]|[21][0-9]|3[10])(/([0-9]+))?([^/0-9]|$)

8. So, let us put the pcre version of regexp with the Emacs version of the regexp. Yes, the pcre version is shorter, and has lesser number of backslash-es. (The pcre version is not very readable, but it is much more readable than the Emacs version)

PCRE-version of above elisp regexp

^ *(0?[1-9]|1[210])/(0?[1-9]|[21][0-9]|3[10])(/([0-9]+))?([^/0-9]|$)

Emacs-version of above elisp regexp

^ *\(0?[1-9]\|1[012]\)/\(0?[1-9]\|[12][0-9]\|3[01]\)\(/\([0-9]+\)\)?\([^/0-9]\|$\)

9. Let us do a regexp-replace in PCRE-mode. We will be converting the American-style dates to British / Indian-style dates.

pcre-query-replace-regexp (C-c / %)

and enter the following for regexp,

^ *(0?[1-9]|1[210])/(0?[1-9]|[21][0-9]|3[10])(/([0-9]+))?([^/0-9]|$)

and following for replacement

\2-\1\,(if \4 (concat "-" \4) "")

10. Once the replacement is done your buffer will look like this

;; American style dates for matching the regexp
30-5
30-5-7
"^ *\\(0?[1-9]\\|1[012]\\)/\\(0?[1-9]\\|[12][0-9]\\|3[01]\\)\\(/\\([0-9]+\\)\\)?\\([^/0-9]\\|$\\)"

Yes, the replacement works.

11. Now put the cursor on the regexp, and do the following

(When I did this my Emacs (emacs -Q) crashed. So, I am restarting Emacs for the following screenshots)

12. This is what the above command does. It breaks down the elisp regexp in to its component parts .

Result of rxt-explain

(In the above screenshot, rxt-help-mode-map shows entries for its parent map elisp-mode as well. I have slightly modified the menu snippet, after taking the screenshot)