How do I break a large piece of text in to a list of 5-word phrases, and identify the oft-repeated phrases [closed]

Question

I would like a function that finds repeating sequences of text sections. Would start with defining a number N. Let I have (setq N 5) say. I would take the first five text sections, then look for matches of the five text sections, by taking the next five text sections (a shift by one text section) as input. Shifting by one text section each time. Then start again, taking the next five text sections from the beginning, this time shifted by one text section, and repeat again.

Here is an example text

Joseph Rudyard Kipling (30 December 1865 - 18 January 1936)
was an English novelist, short-story writer, poet, and
journalist. He was born in British India, which inspired
much of his work.  English novelist, short-story writer,
poet, and journalist.

Kipling's works of fiction include the Jungle Book duology
(The Jungle Book, 1894; The Second Jungle Book, 1895), Kim
(1901), the Just So Stories (1902) and many short stories,
including "The Man Who Would Be King" (1888).  His poems
include "Mandalay" (1890), "Gunga Din" (1890), "The Gods of
the Copybook Headings" (1919), "The White Man's Burden"
(1899), and "If-" (1910).  He is seen as an innovator in
the art of the short story.  His children's books are
classics.

---

Illustration

Let N = 5, the function would take the first five text collections as the Search Sequence.

Joseph Rudyard Kipling (30 December

A search for a match is performed on the rest of the text by getting text shifted by one text section at a time to obtain the Text Extract.

Joseph Rudyard Kipling (30 December
Rudyard Kipling (30 December 1865
Kipling (30 December 1865 -

and so on, until reaching the end of the buffer.

Then the Search Sequence is updated to

Rudyard Kipling (30 December 1865

and the process is repeated.

---

Illustration

Let us now arrive at the Search Sequence

novelist, short-story writer, poet, and 

we take the Text Extracts

Joseph Rudyard Kipling (30 December
Rudyard Kipling (30 December 1865
Kipling (30 December 1865 -
(30 December 1865 - 18
December 1865 - 18 January

eventually reaching the match with

novelist, short-story writer, poet, and 

And we continue to the end of the buffer.

---

Output

The output would be the line number where the match starts followed by the Search Sequence

4- novelist, short-story writer, poet, and 

Answer 1

In response to your request on my first answer, which I would like to preserve (so that users can compare the answers), I am posting here the modified version to get all repeated patterns in one go:

(require 'thingatpt)

(defun get-pattern ()
  "Get list of 5 words.
Return nil if no 5 words left."
  (let (words)
    (dotimes (_ 5)
      (if (forward-word)
          (push (word-at-point t) words)
        (setq words nil)))
    words))

(defun scan-buffer-for-pattern ()
  (let ((x (get-pattern))
        (y t) ;when y becomes nil we have reached eob
        matches)
    (while y ;if match and not eob, return match
      (save-excursion
        (setq y (get-pattern))) ;nil if eob
      (when (and x (equal x y))
        (let ((l (line-number-at-pos)))
          (push (cons (+ l (if (/= l (save-excursion (forward-word)
                                                     (line-number-at-pos)))
                               1 0)) ;fix line number if pattern on new line
                    (mapconcat #'identity (reverse x) " "))
              matches)))
      (forward-word))
    (nreverse matches)))

(defun find-repeated-pattern ()
  (interactive)
  (goto-char (point-min))
  (let (matches)
    (while (not (eobp)) ;self explanatory
      (save-excursion ;jump back if no match
        (push (scan-buffer-for-pattern) matches))
      (forward-word))
    (print (delete nil (nreverse matches)))))

You could add some extra (e.g. twice) repeated patterns to the example text, to see why the elements in the returned list are lists of conses.

Simply do M-x find-repeated-pattern; for this case, I have simply added the goto-char statement to the start of that function.

Answer 2

To start: This question might need more detail/clarification. On my first reading of the question, I believed the OP was asking for a general search function (ex. If find or regex was fed a sequence of 5 words, it would display all the matches.)

If I do not interpret the desired as a general search, then I believe you are asking for the function, given N as input, to iteratively:

1. If there are N words at point and ahead, take them as input. If not, return output and exit.
2. Capture the search results of this input within the document as output.
3. Move forward one word.

(In other words, something similar to what writing assistant programs do to improve writing quality by reducing repeated phrases. We don't know what said phrases are ahead of time.)

I am unsure of what the desired output is. For simplicity, the output is a set of marks pushed to the mark ring. This way, you can use the mark ring to iterate through the results for editing/viewing as is best suited to whichever use case you have.

We can implement a function like this using regex to search for each phrase with words N:

(defun search-repeating-N-words ()
"Search buffer for repeating phrases with N number of words."
  (interactive)
  (let (N x regex-search)
    (setq N (read-number "How many words to search?: " 5))
    (setq x 1)
    (save-excursion
      ;; The search at the end will execute even if there aren't N words left.
      ;; Given that it only searches forward, this is inefficient but not erroneous.
      (while (< x (length (buffer-string))
                (save-excursion
                        (let (p1 p2 (case-fold-search t))
                          (setq p1 x)
                          ;; After search N words forward,
                          ;; set end point as index of the last char of those words
                          (dotimes (y N (setq p2 (point)))
                              (skip-chars-forward "_a-z0-9"))

                          (setq regex-search
                             (buffer-substring-no-properties p1 p2)))

                      ;; Only forward search is necessary. If it was repeated behind,
                      ;; we would have caught it in the previous iterations.
                      ;; For the record, this implementation does introduce
                      ;; the problem where the same repeated phrase is
                      ;; captured by multiple earlier searches.
                      (save-excursion
                        (while (search-forward regex-search nil t)
                          (let (p2)
                           (setq p2 (point))
                           (goto-char (- p2 (length regex-search)))
                           (push-mark p2))))

                      (setq x (+ x (skip-chars-forward "_a-zA-Z0-9") 1))))))))

Answer 3

I am typing as I work out what you want.

This is the Rudyard Kipling text you started with ...

Rudyard Kipling (30 December 1865 - 18 January 1936)
was an English novelist, short-story writer, poet, and 
journalist. He was born in British India, which inspired 
much of his work.  English novelist, short-story writer, 
poet, and journalist.

Kipling's works of fiction include the Jungle Book duology 
(The Jungle Book, 1894; The Second Jungle Book, 1895), Kim 
(1901), the Just So Stories (1902) and many short stories, 
including "The Man Who Would Be King" (1888).  His poems 
include "Mandalay" (1890), "Gunga Din" (1890), "The Gods of 
the Copybook Headings" (1919), "The White Man's Burden" 
(1899), and "If-" (1910).  He is seen as an innovator in 
the art of the short story.  His children's books are 
classics. 

You are interested only in words. So replace all non-word characters with SPC.

C-M-% [^[:word:]] RET <SPC> RET

You will get

Rudyard Kipling  30 December 1865   18 January 1936  was an English novelist  short story writer  poet  and  journalist  He was born in British India  which inspired  much of his work   English novelist  short story writer   poet  and journalist   Kipling's works of fiction include the Jungle Book duology   The Jungle Book  1894  The Second Jungle Book  1895   Kim   1901   the Just So Stories  1902  and many short stories   including  The Man Who Would Be King   1888    His poems  include  Mandalay   1890    Gunga Din   1890    The Gods of  the Copybook Headings   1919    The White Man's Burden    1899   and  If    1910    He is seen as an innovator in  the art of the short story   His children's books are  classics   

After that is done, go to the end of your text, and add two newlines. (You can see the extra newline, I have added at the end of the text)

Now you define a macro which does this

1. Copy the first 5 words of the buffer, in to the end of the buffer
2. ... and deletes the first word

That is the sequence would roughly look like

(defalias 'take-five-and-shift-left
   (kmacro "<escape> < C-SPC C-u 5 M-f <escape> w <escape> > C-x C-k <tab> . SPC C-y C-q C-j <escape> < M-d" 1 "%d"))

Now at the end of this step, you get this

Kipling  30 December 1865   18 January 1936  was an English novelist  short story writer  poet  and  journalist  He was born in British India  which inspired  much of his work   English novelist  short story writer   poet  and journalist   Kipling's works of fiction include the Jungle Book duology   The Jungle Book  1894  The Second Jungle Book  1895   Kim   1901   the Just So Stories  1902  and many short stories   including  The Man Who Would Be King   1888    His poems  include  Mandalay   1890    Gunga Din   1890    The Gods of  the Copybook Headings   1919    The White Man's Burden    1899   and  If    1910    He is seen as an innovator in  the art of the short story   His children's books are  classics   

0. Rudyard Kipling  30 December 1865

Note that the first word has been deleted from the beginning of the buffer, and the first 5 words are copied to the end of the file.

Now keep repeating the macro with C-x e e e e ... etc till you reach the end of the text.

I repeated it for many times, and this is what I get

Stories  1902  and many short stories   including  The Man Who Would Be King   1888    His poems  include  Mandalay   1890    Gunga Din   1890    The Gods of  the Copybook Headings   1919    The White Man's Burden    1899   and  If    1910    He is seen as an innovator in  the art of the short story   His children's books are  classics   

0. Rudyard Kipling  30 December 1865
1. Kipling  30 December 1865   18
2. 30 December 1865   18 January
3. December 1865   18 January 1936
4. 1865   18 January 1936  was
5. 18 January 1936  was an
6. January 1936  was an English
7. 1936  was an English novelist
8. was an English novelist  short
9. an English novelist  short story
10. English novelist  short story writer
11. novelist  short story writer  poet
12. short story writer  poet  and
13. story writer  poet  and  journalist
14. writer  poet  and  journalist  He
15. poet  and  journalist  He was
16. and  journalist  He was born
17. journalist  He was born in
18. He was born in British
19. was born in British India
20. born in British India  which
21. in British India  which inspired
22. British India  which inspired  much
23. India  which inspired  much of
24. which inspired  much of his
25. inspired  much of his work
26. much of his work   English
27. of his work   English novelist
28. his work   English novelist  short
29. work   English novelist  short story
30. English novelist  short story writer
31. novelist  short story writer   poet
32. short story writer   poet  and
33. story writer   poet  and journalist
34. writer   poet  and journalist   Kipling's
35. poet  and journalist   Kipling's works
36. and journalist   Kipling's works of
37. journalist   Kipling's works of fiction
38. Kipling's works of fiction include
39. works of fiction include the
40. of fiction include the Jungle
41. fiction include the Jungle Book
42. include the Jungle Book duology
43. the Jungle Book duology   The
44. Jungle Book duology   The Jungle
45. Book duology   The Jungle Book
46. duology   The Jungle Book  1894
47. The Jungle Book  1894  The
48. Jungle Book  1894  The Second
49. Book  1894  The Second Jungle
50. 1894  The Second Jungle Book
51. The Second Jungle Book  1895
52. Second Jungle Book  1895   Kim
53. Jungle Book  1895   Kim   1901
54. Book  1895   Kim   1901   the
55. 1895   Kim   1901   the Just
56. Kim   1901   the Just So
57. 1901   the Just So Stories
58. the Just So Stories  1902
59. Just So Stories  1902  and
60. So Stories  1902  and many

Please confirm that I have understood you correctly.

I am assuming that you are reasonably good at creating keyboard macros. You need not be a programmer at all.

Answer 4

This is the second take

1. Create a file called kipling.txt with following contents.

Rudyard Kipling (30 December 1865 - 18 January 1936)
was an English novelist, short-story writer, poet, and 
journalist. He was born in British India, which inspired 
much of his work.  English novelist, short-story writer, 
poet, and journalist.

Kipling's works of fiction include the Jungle Book duology 
(The Jungle Book, 1894; The Second Jungle Book, 1895), Kim 
(1901), the Just So Stories (1902) and many short stories, 
including "The Man Who Would Be King" (1888).  His poems 
include "Mandalay" (1890), "Gunga Din" (1890), "The Gods of 
the Copybook Headings" (1919), "The White Man's Burden" 
(1899), and "If-" (1910).  He is seen as an innovator in 
the art of the short story.  His children's books are 
classics. 

2. Copy the following snippets to *scratch*, and do M-x eval-buffer.

(defalias 'normalize-text
   (kbd "<escape> < C-x h C-x C-l ! C-M-% [ ^ [ : w o r d : ] ] <return> SPC <return> ! <escape> < C-M-% SPC + <return> SPC <return> ! <return> <return> C-x C-s"))

(defalias 'take-five-and-shift-left
   (kbd "<escape> < C-SPC C-u 5 M-f <escape> w <escape> > C-q C-j C-y <escape> < M-d C-d"))

(defun highlight-repeating-lines ()
  (interactive)
  (hi-lock-mode 1)
  ;; ^\(.+
  ;; \)\1+
  (highlight-regexp (rx (and (group bol (one-or-more any) "\n") (one-or-more (backref 1)) )) 'highlight)
  (font-lock-flush))

(global-set-key (kbd "C-c C-1") 'normalize-text)
(global-set-key (kbd "C-c C-c") 'take-five-and-shift-left)
(global-set-key (kbd "C-c C-h") 'highlight-repeating-lines)

3. Now go to the beginning of buffer kipling.txt and C-c C-1.

You will text that is stripped of non-word characters, has no extra spaces, and is downcased.

That is you will get this

rudyard kipling 30 december 1865 18 january 1936 was an english novelist short story writer poet and journalist he was born in british india which inspired much of his work english novelist short story writer poet and journalist kipling's works of fiction include the jungle book duology the jungle book 1894 the second jungle book 1895 kim 1901 the just so stories 1902 and many short stories including the man who would be king 1888 his poems include mandalay 1890 gunga din 1890 the gods of the copybook headings 1919 the white man's burden 1899 and if 1910 he is seen as an innovator in the art of the short story his children's books are classics

4. Go to the beginning of buffer, and keep pressing C-c C-c until you exhaust the text. (Use a large prefix argument like C-u 100 C-c C-C or some such thing to repeat the command multiple times)

5. You will be left with the following text

rudyard kipling 30 december 1865
kipling 30 december 1865 18
30 december 1865 18 january
december 1865 18 january 1936
1865 18 january 1936 was
18 january 1936 was an
january 1936 was an english
1936 was an english novelist
was an english novelist short
an english novelist short story
english novelist short story writer
novelist short story writer poet
short story writer poet and
story writer poet and journalist
writer poet and journalist he
poet and journalist he was
and journalist he was born
journalist he was born in
he was born in british
was born in british india
born in british india which
in british india which inspired
british india which inspired much
india which inspired much of
which inspired much of his
inspired much of his work
much of his work english
of his work english novelist
his work english novelist short
work english novelist short story
english novelist short story writer
novelist short story writer poet
short story writer poet and
story writer poet and journalist
writer poet and journalist kipling's
poet and journalist kipling's works
and journalist kipling's works of
journalist kipling's works of fiction
kipling's works of fiction include
works of fiction include the
of fiction include the jungle
fiction include the jungle book
include the jungle book duology
the jungle book duology the
jungle book duology the jungle
book duology the jungle book
duology the jungle book 1894
the jungle book 1894 the
jungle book 1894 the second
book 1894 the second jungle
1894 the second jungle book
the second jungle book 1895
second jungle book 1895 kim
jungle book 1895 kim 1901
book 1895 kim 1901 the
1895 kim 1901 the just
kim 1901 the just so
1901 the just so stories
the just so stories 1902
just so stories 1902 and
so stories 1902 and many
stories 1902 and many short
1902 and many short stories
and many short stories including
many short stories including the
short stories including the man
stories including the man who
including the man who would
the man who would be
man who would be king
who would be king 1888
would be king 1888 his
be king 1888 his poems
king 1888 his poems include
1888 his poems include mandalay
his poems include mandalay 1890
poems include mandalay 1890 gunga
include mandalay 1890 gunga din
mandalay 1890 gunga din 1890
1890 gunga din 1890 the
gunga din 1890 the gods
din 1890 the gods of
1890 the gods of the
the gods of the copybook
gods of the copybook headings
of the copybook headings 1919
the copybook headings 1919 the
copybook headings 1919 the white
headings 1919 the white man's
1919 the white man's burden
the white man's burden 1899
white man's burden 1899 and
man's burden 1899 and if
burden 1899 and if 1910
1899 and if 1910 he
and if 1910 he is
if 1910 he is seen
1910 he is seen as
he is seen as an
is seen as an innovator
seen as an innovator in
as an innovator in the
an innovator in the art
innovator in the art of
in the art of the
the art of the short
art of the short story
of the short story his
the short story his children's
short story his children's books
story his children's books are
his children's books are classics
children's books are classics

6. Do C-x h M-x sort-lines. and you will get this

18 january 1936 was an
1865 18 january 1936 was
1888 his poems include mandalay
1890 gunga din 1890 the
1890 the gods of the
1894 the second jungle book
1895 kim 1901 the just
1899 and if 1910 he
1901 the just so stories
1902 and many short stories
1910 he is seen as
1919 the white man's burden
1936 was an english novelist
30 december 1865 18 january
an english novelist short story
an innovator in the art
and if 1910 he is
and journalist he was born
and journalist kipling's works of
and many short stories including
are classics
art of the short story
as an innovator in the
be king 1888 his poems
book 1894 the second jungle
book 1895 kim 1901 the
book duology the jungle book
books are classics
born in british india which
british india which inspired much
burden 1899 and if 1910
children's books are classics
classics
copybook headings 1919 the white
december 1865 18 january 1936
din 1890 the gods of
duology the jungle book 1894
english novelist short story writer
english novelist short story writer
fiction include the jungle book
gods of the copybook headings
gunga din 1890 the gods
he is seen as an
he was born in british
headings 1919 the white man's
his children's books are classics
his poems include mandalay 1890
his work english novelist short
if 1910 he is seen
in british india which inspired
in the art of the
include mandalay 1890 gunga din
include the jungle book duology
including the man who would
india which inspired much of
innovator in the art of
inspired much of his work
is seen as an innovator
january 1936 was an english
journalist he was born in
journalist kipling's works of fiction
jungle book 1894 the second
jungle book 1895 kim 1901
jungle book duology the jungle
just so stories 1902 and
kim 1901 the just so
king 1888 his poems include
kipling 30 december 1865 18
kipling's works of fiction include
man who would be king
man's burden 1899 and if
mandalay 1890 gunga din 1890
many short stories including the
much of his work english
novelist short story writer poet
novelist short story writer poet
of fiction include the jungle
of his work english novelist
of the copybook headings 1919
of the short story his
poems include mandalay 1890 gunga
poet and journalist he was
poet and journalist kipling's works
rudyard
rudyard kipling
rudyard kipling 30
rudyard kipling 30 december
rudyard kipling 30 december 1865
second jungle book 1895 kim
seen as an innovator in
short stories including the man
short story his children's books
short story writer poet and
short story writer poet and
so stories 1902 and many
stories 1902 and many short
stories including the man who
story his children's books are
story writer poet and journalist
story writer poet and journalist
the art of the short
the copybook headings 1919 the
the gods of the copybook
the jungle book 1894 the
the jungle book duology the
the just so stories 1902
the man who would be
the second jungle book 1895
the short story his children's
the white man's burden 1899
was an english novelist short
was born in british india
which inspired much of his
white man's burden 1899 and
who would be king 1888
work english novelist short story
works of fiction include the
would be king 1888 his
writer poet and journalist he
writer poet and journalist kipling's

(If you notice carefully, I have deleted some short sentences that were sticking out like sore thumb ... this is cheating, but let us live with it)

7. Now we will try to identify the repeated phrases in that text. Wasn't this the objective all along.

• M-x re-builder
• C-c TAB rx
• Copy the following regexp pattern

'(and (group bol (one-or-more any) "\n") (one-or-more (backref 1)) )

The above rx form is equivalent to entering the following at the minibuffer ... and yes, there is a literal newline in there in the regexp.

\(^.+
\)\1+

This is what you will see

Highlight repeating lines with re-builder

8. Kill the re-builder buffer.

9. Now do C-c C-h. You may have to repeat it multiple times, or scroll the buffer up and down couple of times for the font-lock to kick in ...

At the end of this step this is what you will see

Repeated 5-word phrases in a large chunk of text, highlighted with highlight-regexp

Answer 5

This is the elisp version of the keyboard macro version that I shared earlier here https://emacs.stackexchange.com/a/74231/31220

1. C-x C-f kipling.txt
2. Copy the following elisp snippet to *scratch* buffer, and do M-x eval-buffer

Compute N-gram frequency table of a chunk of text

(Dash makes it pretty easy to follow the assembly line. Each step in this assembly line, more or less corresponds to a step in the above indicated answer. Even if you aren't a programmer, it is possible to identify what each step is doing ...)

(require 'dash)
(require 's)

(->> (with-current-buffer "kipling.txt"
       (buffer-substring-no-properties (point-min) (point-max)))
     s-downcase
     (s-replace-regexp (rx (not word)) " ")
     (s-replace-regexp (rx (one-or-more space)) " ")
     s-split-words
     ;; (-unfold (lambda (x) (when (cdr x) (cons (-take 5 x) (cdr x)))))
     (-partition-in-steps 5 1)
     (--map (s-join " " it))
     (-group-by #'identity)
     (-sort (-on #'> (-on 'length 'cdr)))
     (--keep (when (> (length (cdr it)) 1)
               (list (car it)
                     (number-to-string (length (cdr it))))))
     (--map (s-join "|" it))
     (--map (concat "\n| " it))
     (s-join "")
     ((lambda (s)
        (with-current-buffer (get-buffer-create "*ngram-frequency*")
          (erase-buffer)
          (insert s)
          (org-mode)
          (org-table-align)
          (goto-char (point-min))
          (pop-to-buffer (current-buffer))))))

3. You will get a pop-up buffer with the following table

| english novelist short story writer | 2 |
| novelist short story writer poet    | 2 |
| short story writer poet and         | 2 |
| story writer poet and journalist    | 2 |

Answer 6

As it looked to me that it would be handy to use word-at-point (or symbol-at-point) and forward-word (or (forward-symbol 1) if you don't like to break words separated by 'dashes') in the solution, I have coded the following alternative, and from my perspective, most straightforward, solution. Although the solution is quite self-explanatory if read carefully, I've used indicative function names and added a docstring and some comments for extra explanation

(require 'thingatpt)

(defun get-pattern ()
  "Get list of 5 words.
Return nil if no 5 words left."
  (let (words)
    (dotimes (_ 5)
      (if (forward-word)
          (push (word-at-point t) words)
        (setq words nil)))
    words))

(defun scan-buffer-for-pattern ()
  (let ((x (get-pattern))
        (y t) ;when y becomes nil we have reached eob
        match)
    (while (and y (not match)) ;if match and not eob, return match
      (save-excursion
        (setq y (get-pattern)) ;nil if eob
        (setq match (equal x y)))
      (unless match (forward-word))) ;if not eob, forward word
    (when y ;if not eob, return point
      (cons (line-number-at-pos) (mapconcat #'identity (nreverse x) " ")))))

(defun find-repeated-pattern ()
  (interactive)
  (let (match)
    (while (and (not match) (not (eobp))) ;self explanatory
      (save-excursion ;jump back if no match
        (setq match (scan-buffer-for-pattern)))
      (unless match (forward-word)))
    (when match
      (print match))))

You should first place the cursor at the beginning of the buffer (or where you'd like to start) and then do M-x find-repeated-pattern. Of course, you could add a goto-char for this at the start of find-repeated-pattern.

I think you will best understand it by first reading my solution (to get the concept) and then trying, with inspiration from my solution, to recreate the solution yourself from scratch (and compare to my solution).

I have written it to return the first repeated pattern found, if you'd like to change the behavior, then I think with some thinking, it will be quite easy to adapt the solution as you like.

Good luck!