What I'm trying to do

I'm trying to condense a set of research notes consisting of over 250,000 words across 50+ files. I've converted these from previous file formats (mostly mind maps and OPML files) into org-mode. There are lots of duplicated notes where I previously created new versions of files to allow me to rework ideas.

I have begun consolidating, but I get the feeling there may be a very emacs method of speeding up this process.

What I've done.

I've tried using org-agenda, helm-swoop, helm-ag-edit, iedit, multiple-cursors, multi-occur, diff, ediff, and compare-windows.

What I've learnt

I've found that the most important aspect I require is context. For example:

  • Most search functions compile a set of matching lines into another buffer. Even if I show the 8 lines around a match, this doesn't often convey enough of the context for me to be able to recognise a duplicate entry.
  • There are lots of duplicate lines that provide subcategories. While the line is indeed a duplicate, its context is not. For example, Types and Issues below:
* Cars
** Types
*** Sedan
*** Station wagon
** Issues
*** Maintenance
*** Running costs
* Pets
** Types
*** Cat
*** Dog
*** Rabbit
** Issues
*** Fur
*** Toilet training

Manually matching up compare-windows has been the best option so far. It has allowed me to find out just how far a duplicate stretches, often including multiple lines. However, using it alone is quite a manual process:

  1. Find the start of a duplicate between two windows.
  2. Place the cursor at the start of each duplicate in each window.
  3. Run compare-windows to find the end of the duplicate region.
  4. Manually select the region just compared in one of the windows and delete it.

With the myriad of emacs search functions, I'm wondering ... can I do better?

The query

What functions, or combination of functions, could help me to fulfil one or more of the following actions?

  1. Find duplicated sections of text.
    • Using either a full line, or start of a line, or a moving window, as the minimum amount of text to match, but then expanding the match as far as possible.
    • potentially only matching the text content and ignoring org-mode heading levels and whitespace differences.
    • potentially ordered from longest duplicated section of text to some shortest amount of duplicated text (maybe 3-6 words minimum??)
    • at a later stage, potentially with the option for fuzzy matching of lines.
  2. Show a list of the matches.
  3. Display duplicated sections of text in side by side windows.
    • potentially highlighting each duplicated section in some way to show not only where the duplicate starts but where it ends.
    • preferably while leaving the list of matches still open in its own window.

I don't care how CPU intensive the search process is as long as when I come to working though the list of matches, emacs is responsive. I'm using Spacemacs.

Does anyone have suggestions?

  • 2
    This is a difficult question to answer in the EMACS SE context: there are lots of requirements and therefore a lot of sub-questions. I would suggest that you post this entire question on the Org mode mailing list and then try to distill best practices from anything that you get (either there OR here). I suspect that if this question ever gets an answer, you will be the one providing it :-). – NickD Apr 14 '20 at 16:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.