2

One can find here various ways to remove all duplicate lines in Emacs by preserving the first occurrence. Also Emacs 24.4 introduces the command delete-duplicate-lines.

I am wondering if there is a similar way to to delete all duplicate words by preserving the first occurrence of each words.

| improve this question | |
  • If you can format your document one word per line, you'd solve the problem. Can you share more details on how the original document is formatted? – wvxvw Aug 2 '15 at 12:53
  • @wvxvw The original text is a very big text file (no particular structure). If there is a simple function which formats the document one word per line I would be glad to learn about it. – Name Aug 2 '15 at 13:19
  • @Name You can use query-replace to replace spaces with newlines and then delete-duplicate-lines. I am assuming that this document is just a list of words. Because if the document has proper meaningful sentences, they will get butchered. – Kaushal Modi Aug 2 '15 at 13:53
  • @kaushalmodi thank you for this simple idea. – Name Aug 2 '15 at 14:41
  • If there is a direct solution without introducing new lines I would be interested. – Name Aug 2 '15 at 14:42
3

The following command (minimally-tested) will collect all of the words in the buffer and display the unique ones (ignoring case) in a new buffer.

(defun show-unique-words (&optional alphabetical)
  "Collect all of the unique words in the current buffer and
display them in a new buffer.  With prefix, alphabetize the
list."
  (interactive "P")
  (let ((buf (buffer-name))
        (new (get-buffer-create "*Unique Words*"))
        (txt (delete-dups (mapcar #'downcase
                                  (split-string (buffer-string)
                                                nil nil
                                                "[^[:alnum:]]+")))))
    (with-current-buffer new
      (delete-region (point-min) (point-max))
      (insert (format "%d unique words in the <%s> buffer:\n\n"
                      (length txt) buf))
      (cl-dolist (word (if alphabetical (sort txt #'string<) txt))
        (insert (concat word "\n"))))
    (pop-to-buffer new)))
| improve this answer | |
  • Thanks I confirm that your works. I should mention that it is a little bit slow. The above comment of @kaushalmodi (a solution suggesting to replace the space with newlines then using delete-duplicate-lines) seems to be faster. But your function can be use to do other thing (e.g. sorting alphabetically). – Name Aug 2 '15 at 17:24
  • By the way let me ask a question. When using your function interactively (with M-x) how to pass the argument alphabetical to show-unique-words. Using M-x show-unique-words doesn't sort alphabetically. I cannot figure out how to pass this argument to this command. – Name Aug 2 '15 at 17:39
  • 1
    @Name: you can pass the prefix argument to the command with C-u M-x show-unique-words. Have a look at the manual and wiki page on the prefix argument for more information. – Dan♦ Aug 2 '15 at 18:50
  • "a little bit slow" - delete-dups is O(n²), using a hash-table to find duplicates (like delete-duplicate-lines does) would probably be faster for large inputs. – npostavs Aug 2 '15 at 19:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.