How to perform a multi-file regex-search with a regexp that encompass several lines?

Question

Scenario: I want to detect all defcustom sexp that have a :set or :initialize keyword in all Emacs Lisp files inside a directory tree.

Searching for newline can be problematic depending of the regexp engine and the tool that uses it. Despite the following, I did not find something that works for several files:

• [^\0] when back references and look-around are allowed. Unfortunately tools like deadgrep do not support them.
• C-q C-j is another way but again several tools will not accept multiline regexp.
• I have been able to use "defcustom [[:ascii:]]+:\(\(type\)\|\(initialize\)\)" with re-builder to validate that this works inside one file. It won't match text that includes non-ASCII characters, but that's probably OK. I guess I could write code that iterate across all files and use that expression or something like that.

I would like to find an Emacs-based tool that would allow me to write a regexp that support identifying text spread across multiple lines and search for that in all files inside a directory tree without having to write my own. Does such an Emacs-based tool exist?

Answer 1

I don't think you want to just use a single regexp to search for defcustoms that have either :set or :initialize, with (or without) any of those things possibly being on different lines. Why? Because a single regexp to find either :set or :initialize, or both, in either order, is complicated.

Instead, you want to use Lisp syntax: find (defcustom but search the text between the beginning of that match and the end of the defcustom sexp. Search it for :set, then search it for :initialize, without worrying about which might come first or what lines they might be on. No complicated regexp.

IOW, I'd suggest writing a function (command, if you like), that does something like this, in a loop for the file contents (starting from (point-min), with that loop being within a loop over the files you want to search (e.g. directory-files or whatever):

1. search-forward for (defcustom.
2. Go to beginning of the match ((match-beginning 0)), and record that position (e.g. in a let var).
3. forward-sexp to get to the end of the (defcustom...) sexp, and record that position.
4. save-restriction and narrow-to-region.
5. search-forward from (point-min) for :set. If found, accumulate the region bounds as one of your search results, and loop for another search. If not found, go to step 6.
6. Do step 5, but with :initialize in place of :set.

If you want to allow for whitespace between the ( and the defcustom, then use re-search-forward with an appropriate regexp that allows for it.