mu4e has the ability to hide duplicates, with the "V" key, but I see no way to delete the messages. I've tried tricks with marking messages in and out of the hide state, but nothing seems to work well. Searching the documentation and googling has turned up nothing.
3 Answers
As far as I know mu and by extension mu4e does not support this and I don't think it ever will because it does not match its design principles. Instead of providing tools to make it easier to organize your inboxes more easily it replaces organizing with searching.
There is some support for explicitly organizing mail and by saving searches you are effectively doing something similar. But the idea is that we receive so much mail that it has become inefficient to manually organize and vet our inboxes. Its not considered an issue that there might be duplicates, boring mail isn't deleted etc. Mu/mu4e is supposed to let us forget about these issues by providing good search.
There are duplicates in your sea of messages but that isn't considered an issue. Only that duplicates might be displayed is considered an issue. And for that there is (setq mu4e-headers-skip-duplicates t)
.
-
Hmmm... well I guess that goes along with the dropping prices of disk drives, so we can all afford to have terabytes of garbage collecting indefinitely. Not a problem. I can write a script to go delete the dups, I just figured there had to be a way to do it within the app. I guess not. Thanks for your answer.– jtgdOct 4, 2014 at 3:56
-
I needed this too, after having accidentally deleted my ~/Maildir/<provider>/INBOX/.mbsyncstate
. That file seems to be a key piece for mbsync
: because it was missing, mbsync
started redownloading all the messages from the IMAP server, even though they were all still proper files in the INBOX/cur
and INBOX/new
folders...
One solution proposed by the author of mu/mu4e is to do
mu find-dups
and some shell tweaking or the --delete
option, but that didn't work for me (installation of mu-guile
failed).
I ended up writing a program in D that would list all duplicate messages in a maildir folder (thousands for me), and remove one of the duplicates using some custom rules. Below are the main steps.
The most important thing to know is that you only need to remove a message file from the file system so that mu
detects it as a deleted message, and so that this deletion gets propagated to the IMAP server. Deleting a message from within mu4e
does exactly that: it deletes the message file and nothing else. There is no need to touch .mbsyncstate
, this file is automatically updated by mbsync
subsequently.
Best is to work offline and to backup your maildir directory first, e.g. using
cp -pR
(put the backup somewhere wheremu
will not index it, i.e., most likely not in~/Maildir
)Scan all the message files in your maildir, and determine their
Message-ID
(this field is what is used to detect duplicates inmu4e
). For this, I used a regular expression:
// #!/usr/bin/env rdmd
import std.stdio;
import std.file;
import std.path;
import std.algorithm;
import std.array;
import std.regex;
import std.conv;
string[string] msgid_db; // key = msgid, value = filename
string[] duplicates;
/**
extract Message-ID from msg_file (message file in maildir format)
*/
string get_msgid(string msg_file)
{
assert(absolutePath(msg_file).isFile);
string msg_text = to!string(read(msg_file));
auto c = matchFirst(msg_text, regex(`^Message-ID: <(?P<msgid>[^>]+>)`, "im")); // Some tags are 'Message-Id' => flag 'i'. Don't add $ to match end of line, some lines have an extra "(added by [email protected])" etc. Rarely, some don't have Message-ID and mu4e adds a fake-msgid.
if(c.empty)
{
stderr.writeln("Warning: message ", msg_file, " has no Message-ID");
return "";
}
else
return c["msgid"];
}
/**
Scan dirname for message files, getting their message-ID, populating `msgid_db`, and filling `duplicates`
*/
void detect_duplicates(string dirname)
{
// scan the directory for msg files (filter out some non-msg files)
foreach(msg; dirEntries(dirname, SpanMode.depth).filter!(msg => msg.isFile).filter!(msg => !canFind([".DS_Store", ".uidvalidity", ".mbsyncstate", ".mbsyncstate.journal", ".mbsyncstate.lock", ".mbsyncstate.new"],baseName(msg.name) )))
{
// stderr.writeln(msg);
string msgid = get_msgid(msg);
if (msgid != "" && msgid !in msgid_db)
msgid_db[msgid] = msg;
else if (msgid != "")
{
duplicates ~= msgid_db[msgid];
duplicates ~= msg;
}
}
assert(duplicates.length % 2 == 0);
}
After running detect_duplicates
above on a folder name (e.g. ~/Maildir/<provider>/INBOX
), the array duplicates
contains an even number of message file names. Note that both message file names are returned sequentially when duplicates are found! When more than one copy is found, a pair of file names is returned for each copy detected. I.e. if the messages 'A'
, 'B'
, and 'C'
are duplicates, and 'A'
came first, duplicates
will contain the sequence [...,'A','B',...,'A','C',...]
.
- The only remaining step is to do something with the detected duplicate messages. Anything is possible, but I chose to remove the duplicated message in a pair whose timestamp recorded in the filename would be older or (if timestamp is identical) whose UID is greater, which simply amounts to removing
'B'
if the names are such that'A' < 'B'
, and removing'A'
otherwise:
void main(string[] args)
{
if(args.length != 2 || !args[1].isDir)
{
stderr.writeln("Usage: ./duplicates.d dirname");
return;
}
stderr.writeln("*** Detecting duplicate messages in '", args[1], "' ***");
detect_duplicates(args[1]);
foreach(i; 0 .. duplicates.length/2)
if(duplicates[2*i] < duplicates[2*i+1])
writeln("rm ", duplicates[2*i+1]);
else
writeln("rm ", duplicates[2*i]);
stderr.writeln("(There were ", duplicates.length, " messages in the duplicate list, i.e., ", duplicates.length/2, " pairs)");
}
I piped the output written to stdout to a file to_delete.sh
, checked it, then added a first line #!/bin/bash
before executing it.
- I then ran
mu4e
offline first to check duplicates didn't appear anymore (using 'Q' to show all, then 'V' to toggle showing duplicates), then ran it again online andmbsync
did its job removing duplicates on the server.
I came up with this pure mu4e
solution. M-x my-mu4e-header-mark-duplicated
will mark the first of each duplication message set (defined by identical :message-id
and :flags
) in *mu4e-headers*
for further actions. Set (setq mu4e-headers-skip-duplicates nil)
as this requires duplicated messages to be visible in the *mu4e-headers*
buffer.
(defvar *my-mu4e-headers-bol-positions* nil)
(defun my-mu4e-headers-bol-positions ()
"Obtain a list of beginning of line positions for *mu4e-headers*.
`*my-mu4e-headers-bol-positions*' is defined globally, as trying to use
let binding and using add-to-list was unsuccessful."
;; list-bol is nil, equivalent to an empty list.
(with-current-buffer (mu4e-get-headers-buffer)
(setq *my-mu4e-headers-bol-positions* nil)
(save-excursion
(goto-char (point-min))
(while (search-forward mu4e~headers-docid-pre nil t)
(add-to-list '*my-mu4e-headers-bol-positions* (line-beginning-position))
;; Need to move to the end of the line to look for the next line
(end-of-line))
(reverse *my-mu4e-headers-bol-positions*))))
;;
(defun my-mu4e-headers-sexps-with-bol ()
"Obtain the message s-expressions for the messages in *mu4e-headers* and extend with bol."
(let ((list-bol (my-mu4e-headers-bol-positions)))
(when list-bol
(with-current-buffer (mu4e-get-headers-buffer)
(seq-map (lambda (bol)
(let ((msg (get-text-property bol 'msg)))
(plist-put msg :bol bol)))
list-bol)))))
;;
(defun my-mu4e-headers-sexps-with-bol-dups ()
"Obtain the sexps for the messages in *mu4e-headers* with duplicated message-id."
;; https://emacs.stackexchange.com/questions/31448/report-duplicates-in-a-list
(thread-last (my-mu4e-headers-sexps-with-bol)
;; Group by :message-id and :flags (to avoid marking messaged handled differently).
(seq-group-by (lambda (sexp) (list (plist-get sexp :message-id)
(plist-get sexp :flags))))
(seq-filter (lambda (al) (> (length al) 2)))))
;;
(defun my-mu4e-headers-bol-dups ()
"Obtain the beginning of line positions for duplicated messages in *mu4e-headers*.
The beginning of line position for the first of each duplicated messages set is retained."
(thread-last (my-mu4e-headers-sexps-with-bol-dups)
;; First of each duplicated messages set.
(seq-map (lambda (al) (cadr al)))
(seq-map (lambda (sexp) (plist-get sexp :bol)))
(seq-sort #'<)))
;;
(defun my-mu4e-header-mark-duplicated ()
"Mark the first of each duplicate messages set in *mu4e-headers* for an action."
(interactive)
(save-excursion
(mapc
(lambda (bol)
(goto-char bol)
(mu4e-headers-mark-for-something))
(my-mu4e-headers-bol-dups))))
(setq mu4e-headers-skip-duplicates t)
.