Delete duplicate lines, with partial match

Question

Sample text:

This is first line.a
This is over_second line.
This is over_fourth line.
This is third line.
This is over_fourth delete it.
This is over_fourth and one more.
This is over_second with another text.

Need to delete lines where partial match occur, i.e if over_second occurs in another line, then whole line should get deleted. So output will be as follows;

This is first line.
This is over_second line.
This is third line.
This is over_fourth line.

I could only come up with: over_\w+ for selecting section of text. But don't know how to recognize duplicates, and delete whole line.

Answer 1

1. Try delete-duplicate-lines, which is part of distributed Emacs.

2. Emacs Wiki page Duplicate Lines might help.

   • It points to a blog post about it.

   • It explains why interactive search-and-replace might not help.

   • It explains how to do it with Lisp, in various ways.

   • It explains how to do it with the UNIX / GNU/Linux command sort or unique.

Answer 2

If you don't need to do this all the time, the quickest solution might be using interactive replacement: query-replace-regexp, bound to C-M-%. Start with point at the top of the buffer.

Note, if you want to delete entire lines, you'll need to include a newline in your regexp. You enter this at the prompt with C-q C-j.

So, for over_second, call C-M-%, then enter the regular expression:

C-qC-j.*over_second.*

This will match an entire line that contains the string over_second, and includes the previous new line.

Then enter the empty string (just type <enter>) for the replacement value.

The first line that matches the regexp should now be highlighted. This is the one you want to keep, so type n to tell Emacs to skip it. The next line will be highlighted. You can delete this by typing y (or <space>).

You can keep typing y until all the matches are deleted, or you can type ! to delete all remaining matches at once.

Answer 3

I could swear this is a duplicate, but I couldn't find it.

Try this:

(defun my-delete-duplicate-matches (regexp)
  "Delete matching lines, except the first instance of each specific match."
  (interactive (list (read-regexp "Regexp: ")))
  (save-restriction
    (when (use-region-p)
      (narrow-to-region (region-beginning) (region-end)))
    (save-excursion
      (goto-char (point-min))
      (let ((matches (make-hash-table :test #'equal)))
        (save-match-data
          (while (re-search-forward regexp nil :noerror)
            (if (not (gethash (match-string 0) matches))
                (puthash (match-string 0) t matches)
              (forward-line 0)
              (delete-region (point) (progn (forward-line 1)
                                            (point))))))))))

Caveats:

• If the same matching text appears twice on the first line in which it is found, the line will be deleted.

• More generally, if a line contains both the first instance of a given match, and a repeat instance of an earlier match, the line will be deleted.

• Multi-line patterns are not supported.