One of the design choices in package.el was to try and make things "simple". Part of this is that package-initialize searches for all the packages that are installed, then tries to figure out which ones of those should be activated (according to pinning, and recency of versions in case where multiple versions of the same package are available), then loads each activate package's <pkg>-autoloads.el file.

So for N installed packages, that means basically reading N <pkg>-pkg.el package-description files and N <pkg>-autoloads.el files. For large Ns, that can become a serious problem. Another potential performance issue is that it will add N elements to load-path, so every time you load Emacs will search through N directories, so each load is slowed down.

There are various ways we can try and speed this up:

• Provide some way to precompute a ~/.emacs.d/elpa/package-initialize.el(c) file which would be the result of concatenating all the right <pkg>-autoloads.el in the right order. Then package-initialize could just load this file when present and skip everything else. You'd then need some way to refresh/flush the package-initialize.el(c) file when packages are added/updated/removed or when you change your package-pinned-packages or your package-load-list. I think this can be done with fairly few changes to the system (the only thing that would really need changing I think is package-initialize so that it can be told to "only-activate" without loading the metadata about available packages).

• Provide some way to build/manipulate super-packages, i.e. package which combine several packages into one (so there's only one element added to load-path, one <pkg>-pkg.el and one <pkg>-autoloads.el loaded). This might provide more difficult to do (because then you can't activate only part of the packages contained in such super-packages, so the dependency/version analysis could be tricky).

The first option above should be pretty easy to implement and would make package-initialize much faster when you have many packages installed. If you're interested in trying this out, feel free to ask me for help.

FWIW, I've just tried to build such a mega-autoloads file "by hand" on my test setup. Results: while package-initialize takes about 0.9s, loading the mega-autoloads.el file takes 0.3s which I can bring down to 0.2s by let-binding load-source-file-function to nil, and to 0.1s by byte-compiling the file. I expected better speed up, to be honest, but it's still worthwhile.

One of the design choices in package.el was to try and make things "simple". Part of this is that package-initialize searches for all the packages that are installed, then tries to figure out which ones of those should be activated (according to pinning, and recency of versions in case where multiple versions of the same package are available), then loads each activate package's <pkg>-autoloads.el file.

So for N installed packages, that means basically reading N <pkg>-pkg.el package-description files and N <pkg>-autoloads.el files. For large Ns, that can become a serious problem. Another potential performance issue is that it will add N elements to load-path, so every time you load Emacs will search through N directories, so each load is slowed down.

There are various ways we can try and speed this up:

• Provide some way to precompute a ~/.emacs.d/elpa/package-initialize.el(c) file which would be the result of concatenating all the right <pkg>-autoloads.el in the right order. Then package-initialize could just load this file when present and skip everything else. You'd then need some way to refresh/flush the package-initialize.el(c) file when packages are added/updated/removed or when you change your package-pinned-packages or your package-load-list. I think this can be done with fairly few changes to the system (the only thing that would really need changing I think is package-initialize so that it can be told to "only-activate" without loading the metadata about available packages).

• Provide some way to build/manipulate super-packages, i.e. package which combine several packages into one (so there's only one element added to load-path, one <pkg>-pkg.el and one <pkg>-autoloads.el loaded). This might provide more difficult to do (because then you can't activate only part of the packages contained in such super-packages, so the dependency/version analysis could be tricky).

The first option above should be pretty easy to implement and would make package-initialize much faster when you have many packages installed. If you're interested in trying this out, feel free to ask me for help.

FWIW, I've just tried to build such a mega-autoloads file "by hand" on my test setup. Results: while package-initialize takes about 0.9s, loading the mega-autoloads.el file takes 0.3s which I can bring down to 0.2s by let-binding load-source-file-function to nil, and to 0.1s by byte-compiling the file. I expected better speed up, to be honest, but it's still worthwhile.

[EDIT] This "mega autoloads" approach is now available in Emacs's master branch (to become Emacs-27 in some distant future). It's controlled by the new package-quickstart variable.

One of the design choices in package.el was to try and make things "simple". Part of this is that package-initialize searches for all the packages that are installed, then tries to figure out which ones of those should be activated (according to pinning, and recency of versions in case where multiple versions of the same package are available), then loads each activate package's <pkg>-autoloads.el file.

So for N installed packages, that means basically reading N <pkg>-pkg.el package-description files and N <pkg>-autoloads.el files. For large Ns, that can become a serious problem. Another potential performance issue is that it will add N elements to load-path, so every time you load Emacs will search through N directories, so each load is slowed down.

There are various ways we can try and speed this up:

• Provide some way to precompute a ~/.emacs.d/elpa/package-initialize.el(c) file which would be the result of concatenating all the right <pkg>-autoloads.el in the right order. Then package-initialize could just load this file when present and skip everything else. You'd then need some way to refresh/flush the package-initialize.el(c) file when packages are added/updated/removed or when you change your package-pinned-packages or your package-load-list. I think this can be done with fairly few changes to the system (the only thing that would really need changing I think is package-initialize so that it can be told to "only-activate" without loading the metadata about available packages).

• Provide some way to build/manipulate super-packages, i.e. package which combine several packages into one (so there's only one element added to load-path, one <pkg>-pkg.el and one <pkg>-autoloads.el loaded). This might provide more difficult to do (because then you can't activate only part of the packages contained in such super-packages, so the dependency/version analysis could be tricky).

The first option above should be pretty easy to implement and would make package-initialize much faster when you have many packages installed. If you're interested in trying this out, feel free to ask me for help.

One of the design choices in package.el was to try and make things "simple". Part of this is that package-initialize searches for all the packages that are installed, then tries to figure out which ones of those should be activated (according to pinning, and recency of versions in case where multiple versions of the same package are available), then loads each activate package's <pkg>-autoloads.el file.

So for N installed packages, that means basically reading N <pkg>-pkg.el package-description files and N <pkg>-autoloads.el files. For large Ns, that can become a serious problem. Another potential performance issue is that it will add N elements to load-path, so every time you load Emacs will search through N directories, so each load is slowed down.

There are various ways we can try and speed this up:

• Provide some way to precompute a ~/.emacs.d/elpa/package-initialize.el(c) file which would be the result of concatenating all the right <pkg>-autoloads.el in the right order. Then package-initialize could just load this file when present and skip everything else. You'd then need some way to refresh/flush the package-initialize.el(c) file when packages are added/updated/removed or when you change your package-pinned-packages or your package-load-list. I think this can be done with fairly few changes to the system (the only thing that would really need changing I think is package-initialize so that it can be told to "only-activate" without loading the metadata about available packages).

• Provide some way to build/manipulate super-packages, i.e. package which combine several packages into one (so there's only one element added to load-path, one <pkg>-pkg.el and one <pkg>-autoloads.el loaded). This might provide more difficult to do (because then you can't activate only part of the packages contained in such super-packages, so the dependency/version analysis could be tricky).

The first option above should be pretty easy to implement and would make package-initialize much faster when you have many packages installed. If you're interested in trying this out, feel free to ask me for help.

FWIW, I've just tried to build such a mega-autoloads file "by hand" on my test setup. Results: while package-initialize takes about 0.9s, loading the mega-autoloads.el file takes 0.3s which I can bring down to 0.2s by let-binding load-source-file-function to nil, and to 0.1s by byte-compiling the file. I expected better speed up, to be honest, but it's still worthwhile.