How do I delete duplicate messages in mu4e?

Question

mu4e has the ability to hide duplicates, with the "V" key, but I see no way to delete the messages. I've tried tricks with marking messages in and out of the hide state, but nothing seems to work well. Searching the documentation and googling has turned up nothing.

Answer 1

As far as I know mu and by extension mu4e does not support this and I don't think it ever will because it does not match its design principles. Instead of providing tools to make it easier to organize your inboxes more easily it replaces organizing with searching.

There is some support for explicitly organizing mail and by saving searches you are effectively doing something similar. But the idea is that we receive so much mail that it has become inefficient to manually organize and vet our inboxes. Its not considered an issue that there might be duplicates, boring mail isn't deleted etc. Mu/mu4e is supposed to let us forget about these issues by providing good search.

There are duplicates in your sea of messages but that isn't considered an issue. Only that duplicates might be displayed is considered an issue. And for that there is (setq mu4e-headers-skip-duplicates t).

Answer 2

I needed this too, after having accidentally deleted my ~/Maildir/<provider>/INBOX/.mbsyncstate. That file seems to be a key piece for mbsync: because it was missing, mbsync started redownloading all the messages from the IMAP server, even though they were all still proper files in the INBOX/cur and INBOX/new folders...

One solution proposed by the author of mu/mu4e is to do

mu find-dups

and some shell tweaking or the --delete option, but that didn't work for me (installation of mu-guile failed).

I ended up writing a program in D that would list all duplicate messages in a maildir folder (thousands for me), and remove one of the duplicates using some custom rules. Below are the main steps.

The most important thing to know is that you only need to remove a message file from the file system so that mu detects it as a deleted message, and so that this deletion gets propagated to the IMAP server. Deleting a message from within mu4e does exactly that: it deletes the message file and nothing else. There is no need to touch .mbsyncstate, this file is automatically updated by mbsync subsequently.

1. Best is to work offline and to backup your maildir directory first, e.g. using cp -pR (put the backup somewhere where mu will not index it, i.e., most likely not in ~/Maildir)

2. Scan all the message files in your maildir, and determine their Message-ID (this field is what is used to detect duplicates in mu4e). For this, I used a regular expression:

// #!/usr/bin/env rdmd

import std.stdio;
import std.file;
import std.path;
import std.algorithm;
import std.array;
import std.regex;
import std.conv;

string[string] msgid_db; // key = msgid, value = filename
string[] duplicates;

/**
   extract Message-ID from msg_file (message file in maildir format)
 */
string get_msgid(string msg_file)
{
    assert(absolutePath(msg_file).isFile);
    string msg_text = to!string(read(msg_file));

    auto c = matchFirst(msg_text, regex(`^Message-ID: <(?P<msgid>[^>]+>)`, "im")); // Some tags are 'Message-Id' => flag 'i'. Don't add $ to match end of line, some lines have an extra "(added by postmaster@smtp.xxx.com)" etc. Rarely, some don't have Message-ID and mu4e adds a fake-msgid.
    if(c.empty)
    {
        stderr.writeln("Warning: message ", msg_file, " has no Message-ID");
        return "";
    }
    else
        return c["msgid"];
}

/**
   Scan dirname for message files, getting their message-ID, populating `msgid_db`, and filling `duplicates`
*/
void detect_duplicates(string dirname)
{
    // scan the directory for msg files (filter out some non-msg files)
    foreach(msg; dirEntries(dirname, SpanMode.depth).filter!(msg => msg.isFile).filter!(msg => !canFind([".DS_Store", ".uidvalidity", ".mbsyncstate", ".mbsyncstate.journal", ".mbsyncstate.lock", ".mbsyncstate.new"],baseName(msg.name) )))
    {
        // stderr.writeln(msg);
        string msgid = get_msgid(msg);
        if (msgid != "" && msgid !in msgid_db)
            msgid_db[msgid] = msg;
        else if (msgid != "")
        {
            duplicates ~= msgid_db[msgid];
            duplicates ~= msg;
        }       
    }
    assert(duplicates.length % 2 == 0);
}

After running detect_duplicates above on a folder name (e.g. ~/Maildir/<provider>/INBOX), the array duplicates contains an even number of message file names. Note that both message file names are returned sequentially when duplicates are found! When more than one copy is found, a pair of file names is returned for each copy detected. I.e. if the messages 'A', 'B', and 'C' are duplicates, and 'A' came first, duplicates will contain the sequence [...,'A','B',...,'A','C',...].

3. The only remaining step is to do something with the detected duplicate messages. Anything is possible, but I chose to remove the duplicated message in a pair whose timestamp recorded in the filename would be older or (if timestamp is identical) whose UID is greater, which simply amounts to removing 'B' if the names are such that 'A' < 'B', and removing 'A' otherwise:

void main(string[] args)
{
    if(args.length != 2 || !args[1].isDir)
    {
        stderr.writeln("Usage: ./duplicates.d dirname");
        return;
    }

    stderr.writeln("*** Detecting duplicate messages in '", args[1], "' ***");
    detect_duplicates(args[1]);

    foreach(i; 0 .. duplicates.length/2)
        if(duplicates[2*i] < duplicates[2*i+1])
            writeln("rm ", duplicates[2*i+1]);
        else
            writeln("rm ", duplicates[2*i]);
    stderr.writeln("(There were ", duplicates.length, " messages in the duplicate list, i.e., ", duplicates.length/2, " pairs)");
}

I piped the output written to stdout to a file to_delete.sh, checked it, then added a first line #!/bin/bash before executing it.

4. I then ran mu4e offline first to check duplicates didn't appear anymore (using 'Q' to show all, then 'V' to toggle showing duplicates), then ran it again online and mbsync did its job removing duplicates on the server.

Answer 3

I came up with this pure mu4e solution. M-x my-mu4e-header-mark-duplicated will mark the first of each duplication message set (defined by identical :message-id and :flags) in *mu4e-headers* for further actions. Set (setq mu4e-headers-skip-duplicates nil) as this requires duplicated messages to be visible in the *mu4e-headers* buffer.

(defvar *my-mu4e-headers-bol-positions* nil)
(defun my-mu4e-headers-bol-positions ()
  "Obtain a list of beginning of line positions for *mu4e-headers*.

`*my-mu4e-headers-bol-positions*' is defined globally, as trying to use
let binding and using add-to-list was unsuccessful."
  ;; list-bol is nil, equivalent to an empty list.
  (with-current-buffer (mu4e-get-headers-buffer)
    (setq *my-mu4e-headers-bol-positions* nil)
    (save-excursion
      (goto-char (point-min))
      (while (search-forward mu4e~headers-docid-pre nil t)
        (add-to-list '*my-mu4e-headers-bol-positions* (line-beginning-position))
        ;; Need to move to the end of the line to look for the next line
        (end-of-line))
      (reverse *my-mu4e-headers-bol-positions*))))
;;
(defun my-mu4e-headers-sexps-with-bol ()
  "Obtain the message s-expressions for the messages in *mu4e-headers* and extend with bol."
  (let ((list-bol (my-mu4e-headers-bol-positions)))
    (when list-bol
      (with-current-buffer (mu4e-get-headers-buffer)
        (seq-map (lambda (bol)
                   (let ((msg (get-text-property bol 'msg)))
                     (plist-put msg :bol bol)))
                 list-bol)))))
;;
(defun my-mu4e-headers-sexps-with-bol-dups ()
  "Obtain the sexps for the messages in *mu4e-headers* with duplicated message-id."
  ;; https://emacs.stackexchange.com/questions/31448/report-duplicates-in-a-list
  (thread-last (my-mu4e-headers-sexps-with-bol)
    ;; Group by :message-id and :flags (to avoid marking messaged handled differently).
    (seq-group-by (lambda (sexp) (list (plist-get sexp :message-id)
                                       (plist-get sexp :flags))))
    (seq-filter (lambda (al) (> (length al) 2)))))
;;
(defun my-mu4e-headers-bol-dups ()
  "Obtain the beginning of line positions for duplicated messages in *mu4e-headers*.

The beginning of line position for the first of each duplicated messages set is retained."
  (thread-last (my-mu4e-headers-sexps-with-bol-dups)
    ;; First of each duplicated messages set.
    (seq-map (lambda (al) (cadr al)))
    (seq-map (lambda (sexp) (plist-get sexp :bol)))
    (seq-sort #'<)))
;;
(defun my-mu4e-header-mark-duplicated ()
  "Mark the first of each duplicate messages set in *mu4e-headers* for an action."
  (interactive)
  (save-excursion
    (mapc
     (lambda (bol)
       (goto-char bol)
       (mu4e-headers-mark-for-something))
     (my-mu4e-headers-bol-dups))))