How to find a newline with a regular expression in query-replace-regexp?

Question

I am trying to use find-name-dired and query-replace-regexp to find and replace some blocks of text across multiple HTML files. (Just what is explained here in the Emacs Manual.)

query-replace-regexp fails to match anything containing a newline.

For example, I want to find all files containing the following:

      <div class="copyright">

It will find <div class="copyright"> by itself, but not when concatenated with the newline as above.

I've tried coding the newline like so:

1. <div class="copyright">^J (inserted with C-q C-j)
2. <div class="copyright">\n
3. <div class="copyright">"\n".

The first fails with no matches. The second and third cause the minibuffer to complain that \n and "\n" don't match newlines in this function.

I know the newlines in my files are line feeds and not carriage returns (as documented here in Xah Emacs), but I've also tried coding my regular expressions with ^M and \r for good measure. No matches.

How can I get query-replace-regexp to match a newline?

Answer 1

OK, this has been driving me nuts every time I've run into it for over a year now and I finally sat down and figured out what is going on... Unfortunately I don't have a fix---only the actual explanation for the behavior the original poster asked about (I'm on Emacs 29.1.50, MacOS, but that's not super-relevant to the main point here)

% g -> dired-mark-files-containing-regexp

When I run find-name-dired, I don't toggle mark on all files and hit Q. I like to mark the files using % g to test that my regexp working as I expect, then do Q on the marked files.

(The fact that % g finds the expected files with new lines in the query, but Q reports no results found is what was driving me bonkers)

When in the list of files produced by find-name-dired, % g runs the command dired-mark-files-containing-regexp.

In the definition, we see the given regexp is being passed directly to Emacs' re-search-forward:

(defun dired-mark-files-containing-regexp (regexp &optional marker-char)
  ...
  (let ((prebuf (get-file-buffer fn)))
                (message "Checking %s" fn)
                ;; For now we do it inside emacs
                ;; Grep might be better if there are a lot of files
                (if (and prebuf (not dired-always-read-filesystem))
                    (with-current-buffer prebuf
                      (save-excursion
                        (goto-char (point-min))
                        (re-search-forward regexp nil t)))
                  (with-temp-buffer
                    (insert-file-contents fn)
                    (goto-char (point-min))
                    (re-search-forward regexp nil t)))))))
     "matching file")))

Matching files are marked.

Q -> dired-do-find-regexp-and-replace

When you have marked files and do Q, this runs dired-do-find-regexp-and-replace. The help for this function states:

REGEXP should use constructs supported by your local `grep' command.

That was my tip-off to what the issue is here, but following further into the details of what dired-do-find-regexp-and-replace does (indenting under a function call describes what the called function does):

• Assigns "from" (the find pattern) and "to" (the replacement).
• Calls (dired-do-find-regexp from) (returns xrefs of matches)
  • Does a bunch of stuff to list files recursively and handle ignored files to produce files
  • Internally relabels the received from pattern regexp
  • Calls (xref-matches-in-files regexp files)
    • This function's documentation says: "See xref-search-program and xref-search-program-alist for how to control which program to use when looking for matches."
      • xref-search-program defaults to "grep" and specifies the variable value must reference a corresponding entry in xref-search-program-alist

grep as xref-search-program is why this doesn't work.

From Gnu Grep's documentation:

How can I match across lines?

Standard grep cannot do this, as it is fundamentally line-based. Therefore, merely using the [:space:] character class does not match newlines in the way you might expect.

With the GNU grep option -z (–null-data), each input and output “line” is null-terminated; see Other Options. Thus, you can match newlines in the input, but typically if there is a match the entire input is output, so this usage is often combined with output-suppressing options like -q

MacOS's default grep doesn't have the -R or -z options. It's easy to install Gnu Grep from homebrew (brew install grep, which must be run as ggrep), but the output is messy and I suspect not the output that Emacs expects to deal with in the context of creating a buffer listing xrefs to matching regions.

I do not want to get into customizing weird variables for parsing xref lines or whatever in order to make this work.

Why it works in a single buffer

query-replace-regexp (C-M-%) doesn't bring grep into it. It just uses Emacs regexp searching, and then calls perform-replace on regions matching the query.

If the initial searching step of dired-do-find-regexp-and-replace (Q on marked files) finds results, it doesn't call query-replace-regexp; it calls xref-query-replace-in-results, which calls xref--query-replace-1, which finally calls perform-replace (but it has been wrapped in a bajillion layers of special behavior and assumptions at this point).

ugrep: A possible solution? (nope)

In looking at xref-search-program-alist, I noticed it contains an entry for ugrep, which I had not used before. Its README file states:

Matches multiple lines with \n or \R in regex patterns, no special options are required to do so!

WARNING: It looks like it would be A PAIN to install ugrep on Windows if you are not setup in subsystem for Linux.

It was easy to install on my Mac with brew install ugrep.

The ugrep README has a section "Using ugrep within Emacs", but following those instructions did not work for me (Emacs version 29.1.50).

Manuel Uberti writes:

Once set, you can just hit C-x p g (project-find-regexp) in your project and let ugrep do his magic.

In my terminal the following works to find the regions I want to fix:

ugrep -rn '^\s+# rubocop:todo Layout\/LineLength\n(\s+#.*?\n)\s+# rubocop:enable Layout\/LineLength\n' .

However, running C-x p g with either of the following input values gets no results:

• '^\s+# rubocop:todo Layout\/LineLength\n(\s+#.*?\n)\s+# rubocop:enable Layout\/LineLength\n'
• ^\s+# rubocop:todo Layout\/LineLength\n(\s+#.*?\n)\s+# rubocop:enable Layout\/LineLength\n

However, if I follow the find-name-dired > t > Q workflow with the same regexps, it still doesn't work:

Note: ‘\n’ here doesn’t match a newline; to do that, type C-q C-j instead Searching… No matches for: ^\s+# rubocop:todo Layout\/LineLength\n(\s+#.*?\n)\s+# rubocop:enable Layout\/LineLength\n

Since one of the features of ugrep is matching newlines, I'd hoped you wouldn't have to use the C-q C-j syntax. Well, I also hoped the search would work. Unfortunately it also didn't work with line breaks entered via C-q C-j:

Searching… No matches for: ^\s+# rubocop:todo Layout\/LineLength (\s+#.*? )\s+# rubocop:enable Layout\/LineLength

SO, I'm at writing a quick script to deal with this for now, but there's the explanation why % g multiline works in the find-name-dired buffer, but Q with the same query does not.

Answer 2

First, note that query-replace-regexp matches either from the point to the end of the buffer or in the active region. So if your point is after what you are trying to match it will not work.

With that out of the way, the following works for me:

M-x query-replace-regexp RET <div class="copyright"> C-q C-j RET test RET

And you should get a match. C-q simply inserts whatever character you give it. Just pressing C-j would have been interpreted as RET by Emacs.