What would be the best (simplest, with minimal latency) way to load a large dictionary file (about 2M terms; plain text, one term per line), and then do (repeatedly, on a keystroke) a fuzzy search, with the candidates displayed inline?

The search can be started after the fist search token has been typed (like with completion-at-point-functions) or ab initio.

Something like fzf or ivy? But I cannot figure out how to read plain text from a particular file and keep it in memory.

Clarification: there is a ton of packages doing various flavors of completion and presenting the results in the minibuffer (and some also open an annoying helper window at the top of the frame). Those are not the droids I am looking for. I would like to see my completion candidates inline (at point): a dynamic list being displayed immediately under the query as it is typed.


Here's what I would do, using the 466544-strong words.txt file from english-words as an example:

(defun my-dictionary ()
  "Return hash-table whose keys comprise words.txt."
    (insert-file-contents "/path/to/english-words/words.txt")
    (let ((table (make-hash-table :test #'equal :size 466544)))
      (while (not (eobp))
        (puthash (buffer-substring (point) (line-end-position)) nil table)

(defvar my-dictionary
  (lazy-completion-table my-dictionary my-dictionary)
  "Lazy completion table for function `my-dictionary'.")

The basic idea is that you read and cache the contents of your dictionary file into a data structure (a hash-table in my example) which standard completion functions such as completing-read and completion-in-region can understand. The benefit of this is that you can pick and choose between minibuffer and in-buffer completion, as well as from many conforming completion frameworks, using the same completion collection. See (elisp) Completion for more information on this.

After that, you can pass the lazy (cached) collection to whichever conforming completion frontend tickles your fancy. With ivy-mode enabled, for example, a call to

(completion-in-region START END my-dictionary)

for some buffer positions START and END will perform an in-place in-buffer completion like the one you describe.

Here is a more concrete example which completes in-buffer the word preceding point:

(defun my-complete-word-in-region ()
  "Complete word preceding point under `my-dictionary'."
     (skip-syntax-backward "w")

You could then bind this to some convenient key.

If the dictionary file you want to read in does not change very often, you can speed up the initial hash-table-creation latency of the function my-dictionary by creating the hash-table and writing its printed representation to some auxiliary file ahead of time, and then reading that auxiliary file as Lisp instead, though it may not be worth the effort, given a subsequent caching of the hash-table in memory and the inherent latency of fuzzy completion.

See also issue #1174 for some ivy-specific performance tips.

  • ... is there a way to do this starting from something other than a prefix? So it would not be "completion" as such, but general fuzzy search in the dictionary set of strings. Or perhaps starting from a zero-length prefix that matches everything in the dictionary, and then narrowing things down.
    – Dmitri
    Dec 13 '17 at 21:07
  • @Dmitri ivy-mode, as well as other completion frontends such as Helm (maybe Ido and Icicles as well? not sure), already support this. Even the built-in completion-styles aren't restricted to prefix completion, depending on the style used. The built-in styles just aren't as fuzzy as, say, Ivy.
    – Basil
    Dec 13 '17 at 21:29
  • @Dmitri For example, enable ivy-mode and then invoke the sample command my-complete-word-in-region with point placed after some whitespace, i.e. with no word preceding point. This will instigate in-buffer Ivy completion over the entire dictionary, without any prefix. Typing any number of space-delimited substrings, whether prefix, suffix or otherwise, will narrow the completion candidates in a fuzzy manner. Note that Ivy supports multiple types of fuzzy regexp generation, as well as ad-hoc cycling between them.
    – Basil
    Dec 13 '17 at 21:33
  • Yes, this works, thanks for pointing it out! Really cool.
    – Dmitri
    Dec 14 '17 at 2:12

The question is not very clear, to me. But I'm guessing that this might help.

If you use Icicles you can search using various kinds of fuzzy matching.

Icicles search treats search hits as completion candidates, and Icicles lets you use fuzzy matching for completion.

As a perhaps related (unclear from your question) approach, library Synonyms (synonyms.el) uses a large (24MB) thesaurus file. You can query for synonyms of a term, using completion. If you use Icicles then the completion matching can be fuzzy (again, any of several kinds of fuzzy - au choix, changeable anytime while querying/completing).

  • Thanks. Sorry if my question is unclear. If you could name one particular aspect, I could try to improve the question. icicles and Synonyms are relevant but do not fit exactly, as they display candidate matches in the minibuffer only (so far as I can see). I need completion at point, as in youtube.com/watch?v=oMQFqqZeorc and generally similar to modern IDEs.
    – Dmitri
    Dec 10 '17 at 22:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.