I am using filecache as a source to helm-for-files because it ends up being faster than using something like helm-ls-git(EDITED) or helm-projectile. This approach works fine till the cache has around 16000 files. On trying to append another list of around 2000 files, I see emacs memory usage spike until it gets terminated by the OS. At around a file-cache of 16000 file names, the mem usage is around 400mb(that's the gc-limit I have set). On adding these 2000 more files, it hits 10gb and gets killed, since thats the value of ulimit -v. This sounds very much like a bug. Has anyone run into this?

  • Could you use the profiler (M-x profiler-start and M-x profiler-report) to try and see where all that time (and memory) is spent? It's probably a naive algorithmic error.
    – Stefan
    Sep 24 '18 at 20:18

Projectile can cache if you enable it with (setq projectile-enable-caching t). After caching is enabled and after you open the file list with helm-projectile-find-file/projectile-find-file for once, the subsequent invocations gives you the list instanly (~1 second). It works with the Linux kernel source tree with more than 36000 files. My work laptop is using an average hard drive with only 5400 RPM, the MKxx65GSX Hard Disk Drive. Here is its ranking. Not very impressive, as you can see, and the first invocation of helm-projectile-find-file took around ~3 seconds on it, including ignoring files and sorting files.

I tried once with my home directory that contains 100k files and git init it. It took quite a while to retrieve the files, but once the files are retrieved and cached, subsequent invocations of helm-projectile-find-file happens instantly, with no lag when entering pattern and get results immediately. This is just a test and I don't use Projectile with my home directory though.

Finally, there was no memory spike as you suggested because essentially Projectile only stores file list as plain text and does some fancy text processing to manipulate the list.

EDIT: helm-git is an old package (from 2012). If you want real speed, use helm-ls-git that is created from Helm maintainer. It is fast because essentially it only retrieve the list of files from the command git ls-files without fancy processing like removing ignore files and sorting files and create directory list. Took ~1 second on the Linux source tree to for helm-ls-git command to present me the file list. helm-projectile-find-file needs more time because it does more, but you are compensated by its caching.

  • Yes, I was using helm-ls-git. I keep forgetting which ones newer :)
    – Pradhan
    Nov 28 '14 at 7:10
  • So, how things work for you now?
    – Tu Do
    Nov 28 '14 at 7:12
  • About projectile, I followed your guide to using it with helm(brilliant guides btw!) and ran into an issue with helm-projectile on the first run. Never went back to it after.
    – Pradhan
    Nov 28 '14 at 7:13
  • What was the issue may I know?
    – Tu Do
    Nov 28 '14 at 7:15
  • Btw, I tried breaking up that list of 2000 files I mentioned in the question into lists of 200 each. This ended up working. I memory usage suddenly spike up after the filecache became large enough. But after the processing finished, it would fall back. This behaviour does sound like a bug. Will try to report it when possible.
    – Pradhan
    Nov 28 '14 at 7:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.