What I'm trying to do
I'm trying to condense a set of research notes consisting of over 250,000 words across 50+ files. I've converted these from previous file formats (mostly mind maps and OPML files) into org-mode. There are lots of duplicated notes where I previously created new versions of files to allow me to rework ideas.
I have begun consolidating, but I get the feeling there may be a very emacs method of speeding up this process.
What I've done.
I've tried using org-agenda, helm-swoop, helm-ag-edit, iedit, multiple-cursors, multi-occur, diff, ediff, and compare-windows.
What I've learnt
I've found that the most important aspect I require is context. For example:
- Most search functions compile a set of matching lines into another buffer. Even if I show the 8 lines around a match, this doesn't often convey enough of the context for me to be able to recognise a duplicate entry.
- There are lots of duplicate lines that provide subcategories. While the line is indeed a duplicate, its context is not. For example,
* Cars ** Types *** Sedan *** Station wagon ** Issues *** Maintenance *** Running costs * Pets ** Types *** Cat *** Dog *** Rabbit ** Issues *** Fur *** Toilet training
Manually matching up compare-windows has been the best option so far. It has allowed me to find out just how far a duplicate stretches, often including multiple lines. However, using it alone is quite a manual process:
- Find the start of a duplicate between two windows.
- Place the cursor at the start of each duplicate in each window.
- Run compare-windows to find the end of the duplicate region.
- Manually select the region just compared in one of the windows and delete it.
With the myriad of emacs search functions, I'm wondering ... can I do better?
What functions, or combination of functions, could help me to fulfil one or more of the following actions?
- Find duplicated sections of text.
- Using either a full line, or start of a line, or a moving window, as the minimum amount of text to match, but then expanding the match as far as possible.
- potentially only matching the text content and ignoring org-mode heading levels and whitespace differences.
- potentially ordered from longest duplicated section of text to some shortest amount of duplicated text (maybe 3-6 words minimum??)
- at a later stage, potentially with the option for fuzzy matching of lines.
- Show a list of the matches.
- Display duplicated sections of text in side by side windows.
- potentially highlighting each duplicated section in some way to show not only where the duplicate starts but where it ends.
- preferably while leaving the list of matches still open in its own window.
I don't care how CPU intensive the search process is as long as when I come to working though the list of matches, emacs is responsive. I'm using Spacemacs.
Does anyone have suggestions?