How can I improve startup time despite many packages?

Question

TL;DR I have such a huge amount of packages that it's hurting my startup time. If you don't believe that could be the case, read on.

---

My Emacs startup time is quite small. I don't use use-package, I just set tons of hooks and autoloads so that almost all code is deferred. In reality the whole thing is loaded in usually less than half a second, despite it seeming like a crazy mess.

However, over time I noticed that my startup time gets minutely slower, inexplicably. This has eventually gotten to the point where startup time is ≥ 1 second. I finally had enough and I dug into the root of the problem. I eventually commented out my whole ~/.emacs file and found that startup time was still ≥ 1 second. In fact, it had only shaved off ~0.2 seconds, sometimes even less. Then I tried emacs -q and found that the startup time was ~0.1 seconds.

Upon examining this section of the Elisp manual, I found out why emacs -q was reducing startup time so much. Apparently emacs -q stops Emacs from doing three things at startup:

1. loading your init file
2. loading your default.el file
3. calling package-initialize

We've already ruled out my init file, since commenting out my entire ~/.emacs does almost nothing. I don't use a default.el file, so that is also ruled out. Which leaves package-initialize as the culprit for the performance hit.

Why would package-initialize be taking up so much startup time? That was the first question I asked myself. Aren't I autoloading everything? Well, yes. But that is precisely the problem.

I found this post which explains that "activating" packages consists of reading autoload files and setting load paths. This obviously incurs an I/O penalty when you have many packages because you have many autoload files to read and many paths to set. Unfortunately, without this, the task of managing autoloads falls into the hands of the user. In other words, without letting package.el crawl the filesystem for autoload files and paths, I would have to manage that myself which could be a tedious and error-prone process.

I would prefer not to go down that road. I currently have 116 packages, with 107 of those from ELPA and 25 of which are dependencies. I am sure that this whopping number is what is degrading my performance so badly. But I am in a quandary because I do not want to remove any of my packages.

Is there any remedy in such a situation to get my lightning startup time back?

---

Update:

We've started a new thread on the emacs-devel mailing list about some patches by Stefan Monnier (a description of these patches is here) to solve this problem. Anyone is welcome to test out his patches and give feedback.

Another update:

It seems that Stefan Monnier is either not interested in this issue anymore or he is not getting my messages. I am inclined to believe the former, which is fine, though I would appreciate some kind of response from him if that is the case. Anyway, the code which he has produced for this issue so far works quite well. The most recent patches of his can be found here (for Emacs 25.3) and here (for Emacs master branch). I have seen good improvements on my startup time thanks to his patches and I'm at a point where I'm comfortable with my startup time insomuch that it is as optimized as possible without cutting away features of my customization. I was hoping that these patches would make it into the Emacs mainline at some point, but I guess that I (or someone else) would have to take up the torch for it now, instead of Stefan. We had a bit of a spar on the mailing list about copyright assignment and licensing. I was initially uncomfortable with doing so, but due to some comments from Richard Stallman and others, the copyright assignment may not be as restrictive as I originally thought. Moreover, it may be possible for me to commit my works to the public domain as an alternative to copyright assignment. That sounds like the route I will go down if I choose to try to contribute improvements of Stefan's patches to the mainline.

In any case, thanks Stefan for the patches so far! I hope you will continue to develop these changes, but if not, that's okay and I may continue to develop it at some point. I also thank everyone else who offered insight and contribution to solving this issue.

Yet another update:

Wow, looks like this feature finally landed and will be in Emacs 27. Thanks to Stefan Monnier!

Answer 1

One of the design choices in package.el was to try and make things "simple". Part of this is that package-initialize searches for all the packages that are installed, then tries to figure out which ones of those should be activated (according to pinning, and recency of versions in case where multiple versions of the same package are available), then loads each activate package's <pkg>-autoloads.el file.

So for N installed packages, that means basically reading N <pkg>-pkg.el package-description files and N <pkg>-autoloads.el files. For large Ns, that can become a serious problem. Another potential performance issue is that it will add N elements to load-path, so every time you load Emacs will search through N directories, so each load is slowed down.

There are various ways we can try and speed this up:

• Provide some way to precompute a ~/.emacs.d/elpa/package-initialize.el(c) file which would be the result of concatenating all the right <pkg>-autoloads.el in the right order. Then package-initialize could just load this file when present and skip everything else. You'd then need some way to refresh/flush the package-initialize.el(c) file when packages are added/updated/removed or when you change your package-pinned-packages or your package-load-list. I think this can be done with fairly few changes to the system (the only thing that would really need changing I think is package-initialize so that it can be told to "only-activate" without loading the metadata about available packages).

• Provide some way to build/manipulate super-packages, i.e. package which combine several packages into one (so there's only one element added to load-path, one <pkg>-pkg.el and one <pkg>-autoloads.el loaded). This might provide more difficult to do (because then you can't activate only part of the packages contained in such super-packages, so the dependency/version analysis could be tricky).

The first option above should be pretty easy to implement and would make package-initialize much faster when you have many packages installed. If you're interested in trying this out, feel free to ask me for help.

FWIW, I've just tried to build such a mega-autoloads file "by hand" on my test setup. Results: while package-initialize takes about 0.9s, loading the mega-autoloads.el file takes 0.3s which I can bring down to 0.2s by let-binding load-source-file-function to nil, and to 0.1s by byte-compiling the file. I expected better speed up, to be honest, but it's still worthwhile.

[EDIT] This "mega autoloads" approach is now available in Emacs's master branch (to become Emacs-27 in some distant future). It's controlled by the new package-quickstart variable.

Answer 2

The problem you describe about package-initialize taking so much time to load is a well known problem. It is also one of the problems that some emacs frameworks try to solve by loading the autoloads manually.

I see two solutions to your problem.

1. Write (or extract from a framework) the functionality to set the paths and load the autoloads of the packages you are interested in.
2. Use a framework that explicitly targets speed. I personally recommend DOOM emacs. With this framework, I'm loading more than 200 packets in about 1 second.

One of the main resons for recommending DOOM emacs is that the framework puts package management outside of emacs. Don't misinterpret me, it's still emacs that is doing the package management, it's just that managing packages is done outside of a standard user session. The philosophy here is: when normally starting emacs, we should be able to assume that all packages are present and can already be loaded. This saves a lot of time. DOOM emacs provides sort of the equivalent of apt-get or pacman for emacs. Once a package is installed, whenever emacs starts it is assumed to be already installed; no questions asked.