For me, one of Emacs' most useful functions is delete-duplicate-lines. I call it this way, which very helpfully leaves blank lines intact:

(defun delete-duplicate-lines-keep-blanks ()
  (delete-duplicate-lines (region-beginning) (region-end) nil nil t)) 

However, I'd like to be queried. How can I direct Emacs to query me, showing me each duplicate line, and giving me the option to delete it or leave it intact?

2 Answers 2


It seems to be too difficult to make the original delete-duplicate-lines behave in the way you want. Here's something that might do the job though:

(defun my/update-lines (bunches pos keep)
  (cl-loop with dec = (if keep 0 1)
           for line being the hash-key of bunches
           using (hash-value positions) do
            (cl-loop for p in positions
                     if (< p pos) collect p
                     else if (> p pos) collect (- p dec))

(defun my/suggest-delete-line (line)
  (let ((len (length line)))
    (move-overlay selection (point) (+ (point) len))
    (let* ((inhibit-quit t)
               (format "Delete '%s%s'? [y]es/[n]o"
                       (substring line 0 (min len 13))
                        ((> len 16) "...")
                        ((> len 13) (substring line 13 len))
                        (t "")))))))
      (when (= answer ?y)
           (move-end-of-line 1)

(defun my/delete-duplicate-lines (beg end)
   (if (region-active-p)
       (list (region-beginning) (region-end))
     (list (point-min) (point-max))))
  (let ((ignore-white (< (prefix-numeric-value current-prefix-arg) 1))
        (ignore-blank (< (prefix-numeric-value current-prefix-arg) 4))
        (bunches (make-hash-table :test 'equal))
        (selection (make-overlay 1 1)))
    (overlay-put selection 'face 'secondary-selection)
      (goto-char beg)
      (move-beginning-of-line 1)
      (cl-loop for lnum = (count-lines (point-min) beg)
               then (1+ lnum)
               for line = (buffer-substring-no-properties
                             (move-end-of-line 1)
               while (< (point) end) do
                   (or (and (string-match "[ \t]+" line) ignore-white)
                       (and (string-match "^$" line) ignore-blank))
                 (puthash line (cons lnum (gethash line bunches)) bunches))))
    (cl-loop for line being the hash-key of bunches 
             using (hash-value positions)
             unless (cdr positions) do
             (remhash line bunches))
    (cl-loop named :outer for line being the hash-key of bunches do
             (cl-loop for positions = (gethash line bunches)
                      while positions do
                      (cl-loop with continue = t
                               for pos in positions
                               while continue do
                               (goto-char (point-min))
                               (forward-line pos)
                               (cl-case (my/suggest-delete-line line)
                                 (?\C-g (cl-return-from :outer))
                                 (otherwise (setf continue nil)))
                               (my/update-lines bunches pos continue))))
    (delete-overlay selection)))

Which, certainly, could be improved, but at the first glance seems to do what you want.

  • Thanks! I tried this, answering yes to the queries, and it deleted all kinds of lines that weren't duplicates. Feb 17, 2016 at 4:03
  • @incandescentman can you give an example file where it deletes non-duplicate lines?
    – wvxvw
    Feb 17, 2016 at 6:08
  • PS: I've reworked the code to ignore blank and all-whitespace lines by default.
    – wvxvw
    Feb 17, 2016 at 11:50
  • 1
    @incandescentman I do not know whether you got my version running already. But i think, the best way to inform the user of the region to be deleted is to mark it by some distinct text property. I use the face for the secondary selection for that purpose.
    – Tobias
    Feb 18, 2016 at 22:17
  • 1
    @incandescentman It is already implemented in the code of my answer. Ah, it is not text properties, it is an overlay. Sorry for that. Just search for overlay-put.
    – Tobias
    Feb 18, 2016 at 22:23

The following code advices delete-duplicate-lines to get what you want but retains as much of the behavior of the original version as possible.

EDIT: Note, that delete-region is replaced by the corresponding subroutine in the compiled lisp file sort.elc. So we need to reload the source code sort.el.gz to advice delete-region within delete-duplicate-lines.

Make sure that you have have emacs24-el installed on your system.

If you have sort.el instead of sort.el.gz you should adapt the load-library line in the code. You can detect where delete-duplicate-lines is defined by C-h f delete-duplicate-lines. If it just says compiled lisp function but does not give you a source file name then you have only the byte compiled file sort.elc. If you wanted to try the code below you need to get the corresponding source in some way. If the help page tells you that delete-duplicate-line is defined in sort.el, then click the corresponding link. It may be that the link leads you to sort.el or sort.el.gz anyway. In this case you are good with the code below.

  (require 'cl-macs))

(defun delete-duplicate-lines-interactive (oldfun &rest args)
  "Make `delete-region' interactive in `delete-duplicate-lines'."
  (let ((reverse (nth 2 args))
    (adjacent (nth 3 args))
    (arg-interactive (nth 5 args)))
    (if arg-interactive
    (let ((ol (make-overlay 1 1)))
      (overlay-put ol 'face 'secondary-selection) ;; could be customizable
          (let (continue
            (deleted-lines 0))
        (catch :quit
          (advice-add 'delete-region
                  (lambda (delete-region-original start end)
                (if continue
                    (funcall delete-region-original start end)
                  (move-overlay ol start end)
                  (cl-case  (let (mark-active) (read-key "Delete line? ([y]es, [n]ext, [!] all, [q]uit, any other key is equivalent to next):"))
                     (funcall delete-region-original start end)
                     (setq deleted-lines (1+ deleted-lines)))
                     (setq continue t)
                     (funcall delete-region-original start end)
                     (setq deleted-lines (1+ deleted-lines)))
                     (throw :quit nil))))
                (when reverse (goto-char start)))
                  '((name . interactive)))
          (apply oldfun args))
        (message "Deleted %d %sduplicate line%s%s"
             (if adjacent "adjacent " "")
             (if (= deleted-lines 1) "" "s")
             (if reverse " backward" "")))
        (advice-remove 'delete-region 'interactive))
      (delete-overlay ol))
      (apply oldfun args) ;; non-interactive case

(advice-add 'delete-duplicate-lines :around 

(load-library "sort.el") ;; Somehow `delete-region` is replaced in "sort.elc". Therefore load the source version again.

(defun delete-duplicate-lines-keep-blanks ()
  (delete-duplicate-lines (region-beginning) (region-end) nil nil t t))
  • How does it work? Do I need to somehow add call-interactively to (defun delete-duplicate-lines-keep-blanks () (interactive) (delete-duplicate-lines (region-beginning) (region-end) nil nil t))? Feb 17, 2016 at 21:58
  • 1
    @incandescentman I have been able to reproduce the problem. I did not notice it before since I always debugged the stuff and therefore always loaded the source version of sort.el.gz. This also indicates the workaround. Load the source version of the library sort. I have added it to the code. Note, that the source of sort must be installed in the system. If you are writing elisp yourself this is most likely the case.
    – Tobias
    Feb 18, 2016 at 5:35
  • 1
    @incandescentman I've corrected some minor issues to make the code work with emacs -Q. Happy deleting;-). The only minor issue that remains is the number of deleted duplicates in the final message. The original version of delete-duplicate-lines assumes at this point that all duplicate lines have been deleted.
    – Tobias
    Feb 18, 2016 at 8:54
  • 1
    @incandescentman The small issues mentioned in the last comment are also avoided now. The final message is now generated by the advice.
    – Tobias
    Feb 18, 2016 at 9:12
  • 1
    @incandescentman I have replaced (load-library "sort.el.gz") by (load-library "sort.el"). This should work for all source versions -- compressed and uncompressed ones.
    – Tobias
    Feb 18, 2016 at 22:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.