TL;DR I have such a huge amount of packages that it's hurting my startup time. If you don't believe that could be the case, read on.


My Emacs startup time is quite small. I don't use use-package, I just set tons of hooks and autoloads so that almost all code is deferred. In reality the whole thing is loaded in usually less than half a second, despite it seeming like a crazy mess.

However, over time I noticed that my startup time gets minutely slower, inexplicably. This has eventually gotten to the point where startup time is ≥ 1 second. I finally had enough and I dug into the root of the problem. I eventually commented out my whole ~/.emacs file and found that startup time was still ≥ 1 second. In fact, it had only shaved off ~0.2 seconds, sometimes even less. Then I tried emacs -q and found that the startup time was ~0.1 seconds.

Upon examining this section of the Elisp manual, I found out why emacs -q was reducing startup time so much. Apparently emacs -q stops Emacs from doing three things at startup:

  1. loading your init file
  2. loading your default.el file
  3. calling package-initialize

We've already ruled out my init file, since commenting out my entire ~/.emacs does almost nothing. I don't use a default.el file, so that is also ruled out. Which leaves package-initialize as the culprit for the performance hit.

Why would package-initialize be taking up so much startup time? That was the first question I asked myself. Aren't I autoloading everything? Well, yes. But that is precisely the problem.

I found this post which explains that "activating" packages consists of reading autoload files and setting load paths. This obviously incurs an I/O penalty when you have many packages because you have many autoload files to read and many paths to set. Unfortunately, without this, the task of managing autoloads falls into the hands of the user. In other words, without letting package.el crawl the filesystem for autoload files and paths, I would have to manage that myself which could be a tedious and error-prone process.

I would prefer not to go down that road. I currently have 116 packages, with 107 of those from ELPA and 25 of which are dependencies. I am sure that this whopping number is what is degrading my performance so badly. But I am in a quandary because I do not want to remove any of my packages.

Is there any remedy in such a situation to get my lightning startup time back?


Update:

We've started a new thread on the emacs-devel mailing list about some patches by Stefan Monnier (a description of these patches is here) to solve this problem. Anyone is welcome to test out his patches and give feedback.

Another update:

It seems that Stefan Monnier is either not interested in this issue anymore or he is not getting my messages. I am inclined to believe the former, which is fine, though I would appreciate some kind of response from him if that is the case. Anyway, the code which he has produced for this issue so far works quite well. The most recent patches of his can be found here (for Emacs 25.3) and here (for Emacs master branch). I have seen good improvements on my startup time thanks to his patches and I'm at a point where I'm comfortable with my startup time insomuch that it is as optimized as possible without cutting away features of my customization. I was hoping that these patches would make it into the Emacs mainline at some point, but I guess that I (or someone else) would have to take up the torch for it now, instead of Stefan. We had a bit of a spar on the mailing list about copyright assignment and licensing. I was initially uncomfortable with doing so, but due to some comments from Richard Stallman and others, the copyright assignment may not be as restrictive as I originally thought. Moreover, it may be possible for me to commit my works to the public domain as an alternative to copyright assignment. That sounds like the route I will go down if I choose to try to contribute improvements of Stefan's patches to the mainline.

In any case, thanks Stefan for the patches so far! I hope you will continue to develop these changes, but if not, that's okay and I may continue to develop it at some point. I also thank everyone else who offered insight and contribution to solving this issue.

Yet another update:

Wow, looks like this feature finally landed and will be in Emacs 27. Thanks to Stefan Monnier!

  • Great question. – Drew Jan 26 at 15:24
  • use-package is the way to go for this. – Dodgie Jan 26 at 18:20
  • Not trying to downplay the problem (startup latency is important!), but do consider running emacs as a daemon/server so you only pay for startup cost one time. – GManNickG Jan 26 at 23:29
  • 1
    @GManNickG I do run Emacs as a server. Unfortunately, every now and then I push Emacs too hard or tinker with it too much and a restart is the best way to clean things up. When that occurs, I like my startup time to be optimal. – GDP2 Jan 27 at 14:12
up vote 12 down vote accepted

One of the design choices in package.el was to try and make things "simple". Part of this is that package-initialize searches for all the packages that are installed, then tries to figure out which ones of those should be activated (according to pinning, and recency of versions in case where multiple versions of the same package are available), then loads each activate package's <pkg>-autoloads.el file.

So for N installed packages, that means basically reading N <pkg>-pkg.el package-description files and N <pkg>-autoloads.el files. For large Ns, that can become a serious problem. Another potential performance issue is that it will add N elements to load-path, so every time you load Emacs will search through N directories, so each load is slowed down.

There are various ways we can try and speed this up:

  • Provide some way to precompute a ~/.emacs.d/elpa/package-initialize.el(c) file which would be the result of concatenating all the right <pkg>-autoloads.el in the right order. Then package-initialize could just load this file when present and skip everything else. You'd then need some way to refresh/flush the package-initialize.el(c) file when packages are added/updated/removed or when you change your package-pinned-packages or your package-load-list. I think this can be done with fairly few changes to the system (the only thing that would really need changing I think is package-initialize so that it can be told to "only-activate" without loading the metadata about available packages).

  • Provide some way to build/manipulate super-packages, i.e. package which combine several packages into one (so there's only one element added to load-path, one <pkg>-pkg.el and one <pkg>-autoloads.el loaded). This might provide more difficult to do (because then you can't activate only part of the packages contained in such super-packages, so the dependency/version analysis could be tricky).

The first option above should be pretty easy to implement and would make package-initialize much faster when you have many packages installed. If you're interested in trying this out, feel free to ask me for help.

FWIW, I've just tried to build such a mega-autoloads file "by hand" on my test setup. Results: while package-initialize takes about 0.9s, loading the mega-autoloads.el file takes 0.3s which I can bring down to 0.2s by let-binding load-source-file-function to nil, and to 0.1s by byte-compiling the file. I expected better speed up, to be honest, but it's still worthwhile.

[EDIT] This "mega autoloads" approach is now available in Emacs's master branch (to become Emacs-27 in some distant future). It's controlled by the new package-quickstart variable.

  • Those are some very interesting ideas. The first one sounds more down-to-earth and less of a challenge. The second one is quite interesting, but that sounds like a job for the package.el developers. What kind of advice do you have on starting out with that first option? I'd like to see what I can hammer out with it, since it seems much more workable. – GDP2 Jan 26 at 15:54
  • @GDP2: send me an email (grep for "Monnier" in Emacs's files to find my email). – Stefan Jan 26 at 18:13
  • el-get uses the single autoloads file approach, it basically works most of the time. There's trouble with some packages whose autoloads depend on the location in the filesystem that they are evaluated in. I don't follow what you mean by "in the right order" though, why would order of autoload loading ever matter (I didn't think it was even deterministic for current package.el)? – npostavs Jan 27 at 0:13
  • 1
    @npostavs: Most <pkg>-autoloads.el files only setup autoloads and indeed don't care about ordering, but there's nothing preventing them from doing random other things, and package.el does guarantee that the package on which <pkg> depends will be activated before <pkg> itself. – Stefan Jan 27 at 21:43
  • 1
    Another update on this issue: we've started a new topic on the mailing list here and anyone is free to comment on or test out Stefan's changes. – GDP2 Feb 15 at 23:02

The problem you describe about package-initialize taking so much time to load is a well known problem. It is also one of the problems that some emacs frameworks try to solve by loading the autoloads manually.

I see two solutions to your problem.

  1. Write (or extract from a framework) the functionality to set the paths and load the autoloads of the packages you are interested in.
  2. Use a framework that explicitly targets speed. I personally recommend DOOM emacs. With this framework, I'm loading more than 200 packets in about 1 second.

One of the main resons for recommending DOOM emacs is that the framework puts package management outside of emacs. Don't misinterpret me, it's still emacs that is doing the package management, it's just that managing packages is done outside of a standard user session. The philosophy here is: when normally starting emacs, we should be able to assume that all packages are present and can already be loaded. This saves a lot of time. DOOM emacs provides sort of the equivalent of apt-get or pacman for emacs. Once a package is installed, whenever emacs starts it is assumed to be already installed; no questions asked.

  • Phew, glad to know that this issue isn't only affecting me. Thank you for pointing me to DOOM Emacs. I will have to take a closer look at it and see what I can adapt to my own setup. – GDP2 Jan 26 at 15:12
  • I've been playing around with DOOM emacs (and contributing when I can) for some time now. If you are having problems, feel free to contact me. – UndeadKernel Jan 26 at 15:22

Your Answer

 
discard

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.