Search for "foo bar" ignoring comments and newlines

Question

How do I search for "foo bar" ignoring comments and newlines in emacs? For example, given the tex file fragment

blah blah foo % comment 1
% comment 2
bar blah blah

I'd like to be able to type "foo bar" at the emacs search prompt and have my cursor land at the end of "bar" in the third line.

Qualification: I'm actually using Aquamacs in OS X, but hopefully a good answer for emacs would carry over.

Answer 1

I can fake this via C-hv search-whitespace-regexp

It wouldn't be a full solution unless your comment marker character could never occur in any other context (e.g. the string "100%").

For testing select a window of the buffer with the test text, call M-: and paste the following lisp snippet into the minibuffer.

(progn
  (setq-local search-whitespace-regexp "\\([[:space:]\n]\\|%.*\n\\)+")
  (setq-local isearch-lax-whitespace t))

The following figure shows the matches in an extended test text for an isearch starting at the beginning of the buffer.

Answer 2

If you use library hide-comnt.el then you can hide or ignore comments, including for searching (Isearch or Lisp searching).

If the region is active then the effect is limited to the region.

• Command hide/show-comments-toggle toggles hiding and showing comments.

• Command hide/show-comments hides comments or, with a prefix arg, shows them.

• Macro with-comments-hidden temporarily hides comments while it executes a body of code. You do not see the commented text disappear and reappear – it is effectively just ignored while processing the body. The macro uses hide/show-comments to hide the comments.

Answer 3

If you use library Icicles then you can use Icicles search to do the following:

• Search inside or outside of comments. A comment is a THING. Use M-s M-s t and tell it that the kind of THING you want to search is comment.

• Search inside other kinds of THING, optionally ignoring such THINGs when inside comments. Options icicle-ignore-comments-flag and icicle-hide-whitespace-before-comment-flag control this ignoring. You can toggle this ignoring using C-M-; at any time. When comments are ignored, the candidate things (e.g. sexps) to be searched are only those outside of comments.

• You can use icicle-search-text-property (C-c ") to search the zones of text that do not have a text-property value that you specify (e.g., property face with faces font-lock-comment-face and font-lock-comment-delimiter-face - which means search outside comments.

  This is an example of searching the complement of a set of zones. You can turn this context-complementing feature on and off using C-M-~ (icicle-toggle-search-complementing-domain) during Icicles-search completion.

Answer 4

The following lisp code provides the command isearch-visible. It creates a clone of the current buffer and deletes all invisible text there. All isearch operations in the original buffer are then delegated to the clone.

Therefore, you can use Drew's hide/show-comment command from the package hide-comnt.el to hide the comments in your TeX buffer.

Afterwards you can search the visible text as the invisible text never existed with isearch-visible.

If you set search-whitespace-regexp to [[:space:]\n] and use M-x isearch-toggle-lax-whitespace you can use foo bar as search string with isearch-visible in your test TeX buffer to get what you want.

If you want to try the code just paste it into your init file and restart emacs.

CHANGE-LOG:

2017-12-15, 20:30:

1. bugfix: error wrong argument type nil; expected number or marker

2. bugfix: if the match end was at the beginning of a hidden region the calculation of the end position in the original buffer was wrong. This lead to match intervals that were too long.

(require 'subr-x)

(defvar-local clone-visible-clone nil
  "Visible clone of current buffer if there exists one.")

(defvar-local clone-visible-original nil
  "Original buffer of clone.")

(defun clone-visible-buffer ()
  "Function used as result of `isearch-search-fun-function'.
It searches within `visible-isearch-clone-buffer' instead of the original buffer.
Sets `clone-visible-clone' in the original buffer and `clone-visible-original' in the clone."
  (let* ((original-buf (current-buffer))
         buffer-file-name)
    (setq clone-visible-clone
      (with-current-buffer (clone-buffer)
        (setq clone-visible-original original-buf)
        ;; First scan: put original position into 'clone-visible text property
        (cl-loop for int being the intervals property 'invisible
             if (null (invisible-p (car int)))
             do (put-text-property (car int) (cdr int) 'clone-visible-int int))
        ;; Second scan: Remove invisible regions
        (cl-loop for int being the intervals property 'invisible
             if (invisible-p (car int))
             do (delete-region (car int) (cdr int)))
        (current-buffer)))))

(defsubst clone-visible-assert ()
  "Check whether we have a working clone."
  (cl-assert (buffer-live-p clone-visible-clone)
         nil
         "Visible clone %s defective." clone-visible-clone))


(defun clone-visible-original-position (&optional clone-position)
  "Returns position in original buffer corresponding to CLONE-POSITION in visible clone.
CLONE-POSITION defaults to `point'.
We assume that the clone is current."
  (unless clone-position
    (setq clone-position (point)))
  (when-let ((original-int (get-text-property clone-position 'clone-visible-int))
         (clone-int-start (or (previous-single-property-change clone-position 'clone-visible-int) (point-min))))
    (+ clone-position (- (car original-int) clone-int-start))))

(defun clone-visible-clone-position (original-position)
  "Find position in clone corresponding to ORIGINAL-POSITION in current buffer."
  (clone-visible-assert)
  (and original-position
       (with-current-buffer clone-visible-clone
     (cl-loop
      with original-int
      for int being the intervals property 'clone-visible-int
      if (and (setq original-int (get-text-property (car int) 'clone-visible-int))
          (>= original-position (car original-int))
          (< original-position (cdr original-int)))
      return (+ original-position (- (car int) (car original-int)))
      ))))

(defun clone-visible-search-regexp (regexp &optional bound noerror)
  "Searches the visible clone in place of the original buffer.
Uses `isearch-search-fun-default' for the search."
  (let* ((original-buffer (current-buffer))
         (clone-pt (clone-visible-clone-position (point)))
         (clone-bd (clone-visible-clone-position bound))
         (search-fun (isearch-search-fun-default))
         (original-pt (with-current-buffer clone-visible-clone
                        (goto-char clone-pt)
                        (setq clone-pt (funcall search-fun regexp clone-bd noerror))
                        (when clone-pt
                          (let ((match-data (butlast (match-data t))))
                            (cl-loop for int on match-data by #'cddr do
                                     (setcar int (clone-visible-original-position (car int)))
                                     (setcar (cdr int) (1+ (clone-visible-original-position (1- (cadr int))))))
                            (set-match-data (append match-data (list original-buffer)))
                            (match-end 0))))))
    (goto-char (or original-pt bound (point-max)))
    original-pt))

(defun isearch-visible ()
  "Like `isearch-forward' but only searches the visible part of current buffer."
  (interactive)
  (let ((isearch-search-fun-function
     (lambda ()
       #'clone-visible-search-regexp))
    (clone (clone-visible-buffer)))
    (when (buffer-live-p clone)
      (unwind-protect
      (progn
        (isearch-mode t t nil t)))
      (kill-buffer clone))))

Answer 5

Library Isearch+ lets you do this in several ways.

In particular:

• You can dynamically add a search filter that limits Isearch to zones outside comments. Function isearchp-not-in-comment-or-delim-p does this. You can add a filter while searching by hitting C-z & (command isearchp-add-filter-predicate). When prompted for the filter function, you can enter that function name or you can use completion and choose ~[;+].

  (Completion candidates have abbreviated names, and the full filter-predicate name accompanies them as annotations in *Completions*. For this abbreviation:

  • ~ means complement: NOT.
  • [...] means within.
  • ; means comments.
  • + means include the comment delimiters.

  (You can also just set isearch-filter-predicate to isearchp-not-in-comment-or-delim-p, but dynamic Isearch filtering gives you a quick way to filter on the fly.)

• You can search the text of THINGS of various kind (sexps, lists, defuns, lines, pages, sentences, filenames, strings, comments, xml/html elements, symbols,…), using command isearchp-thing. This is equivalent to using command isearchp-thing-define-contexts, which marks such zones with a text property, and then using isearchp-property-forward.

• When searching THINGS, you can toggle ignoring comments, using C-M-; (command isearchp-toggle-ignoring-comments). You can also hide or show code comments during Isearch, using M-; (command isearchp-toggle-hiding-comments).

• You can search within character-property zones. Example: search within zones having a face text property with a value of font-lock-comment-face or font-lock-string-face. Search overlays or text properties.

  From within Isearch: C-t (or C-M-t for regexp search). First time, or with a prefix argument, you are prompted for the property and its values. See the doc string of command isearchp-property-forward.

• For searching within THINGS or searching within character-property zones, you can do the opposite: search outside the given type of THING or outside the given type of property. You can toggle this complementing (inside/outside) anytime during Isearch using C-M-~.

All of these suffer from what is currently a fundamental limitation of Isearch filtering: search hits have to be entirely within the text that remains after filtering, but search itself matches all of the text. See Emacs bug #27896 for a description of the problem. It would be good for someone to provide a solution.

Answer 6

Call me old fashioned, but I would just use standard emacs regexp search

foo\s-*\(//[^
]
\s-*\)*bar

That’s off the top of my head. I will test it in emacs tomorrow - I don’t have emacs on my iPhone :-(.

I use isearch-forward-re to the exclusion of all else. Standard key binding C-M-s. You might prefer re-search-forward.

Commentary

• \s- → character class whitespace
• \s-* → 0 to infinite whitespace
• \(…\) → group

Within the group:

• // → your comment
• [^␤] (with an actual line break instead of ␤ — type C-q C-j) → any character that is not a newline

And thus

//[^
]*
\s-* 

→ comment to end of line, followed by a newline and whitespace at beginning of line.

All of that in a \(…\) group, *’ed to repeat 0 to infinite times.

The funkiest part is the newlines in the regexp. I get them by typing C-q (quote-char) followed by C-j, since C-j is newline.

If all of the above is in an elisp string, then the backslashes will have to be doubled, and you can use \n for a newline.

In the really old days I might have typed [ blank tab ]* - which is illegible. Now I usually remember to type \s-*. But note that syntax class tables can vary.

Modern emacs uses character classes like [:blank:] for tab and space, or [:space:] for any whitespace including tab and space and newline.

So my original regexp might be more readable as

foo[[:space:]]*\(//\([[:blank:]]*[^[:space:]]*\)*\)*[[:space:]]*bar

The trick here is that I don’t know of any easily readable version of “any character except newline”, so instead I do a group of ( any blanks followed by any non-spaces ), which is equivalent when starred,

Again, untested.

foo\s-*\(//[^
]
\s-*\)*bar