I am using
filecache as a source to
helm-for-files because it ends up being faster than using something like
helm-projectile. This approach works fine till the cache has around 16000 files. On trying to append another list of around 2000 files, I see emacs memory usage spike until it gets terminated by the OS. At around a file-cache of 16000 file names, the mem usage is around 400mb(that's the gc-limit I have set). On adding these 2000 more files, it hits 10gb and gets killed, since thats the value of
ulimit -v. This sounds very much like a bug. Has anyone run into this?
I am using
Projectile can cache if you enable it with
(setq projectile-enable-caching t). After caching is enabled and after you open the file list with
projectile-find-file for once, the subsequent invocations gives you the list instanly (~1 second). It works with the Linux kernel source tree with more than 36000 files. My work laptop is using an average hard drive with only 5400 RPM, the MKxx65GSX Hard Disk Drive. Here is its ranking. Not very impressive, as you can see, and the first invocation of
helm-projectile-find-file took around ~3 seconds on it, including ignoring files and sorting files.
I tried once with my home directory that contains 100k files and
git init it. It took quite a while to retrieve the files, but once the files are retrieved and cached, subsequent invocations of
helm-projectile-find-file happens instantly, with no lag when entering pattern and get results immediately. This is just a test and I don't use Projectile with my home directory though.
Finally, there was no memory spike as you suggested because essentially Projectile only stores file list as plain text and does some fancy text processing to manipulate the list.
helm-git is an old package (from 2012). If you want real speed, use
helm-ls-git that is created from Helm maintainer. It is fast because essentially it only retrieve the list of files from the command
git ls-files without fancy processing like removing ignore files and sorting files and create directory list. Took ~1 second on the Linux source tree to for
helm-ls-git command to present me the file list.
helm-projectile-find-file needs more time because it does more, but you are compensated by its caching.