0

Would it not be possible to modify Emacs package system to store all Emacs Lisp files installed from Elpa-compliant packages inside the same directory?

That would dramatically reduce the number of entries inside Emacs load-path and would speed up Emacs startup time, would it not?

On my system I have:

  • 208 directories in my ~/.emacs.d/elpa directory,
  • 182 directories out of these 208 have only files in them, no sub-directories.

I wrote the following Makefile rules in my build system:

.PHONY: timeit
timeit:
        @printf "***** Running Emacs startup time measurement tests\n"
        @printf "** Report Configuration settings.\n"
        $(EMACS) --batch -L . -l $(EMACS_INIT) -l pel-package.el -f pel-package-info
        @printf "\n"
        @printf "** Time mesaurement:\n"
        time -p $(EMACS) -nw -Q -e kill-emacs
        time -p $(EMACS) -nw -q -e kill-emacs
        time -p $(EMACS) -nw -e kill-emacs

My pel-package-info gathers information about the package I use and prints a report on the number used, the load-path, etc...

When I run it a couple of times (with Emacs 26.3) I get something like this:

> make timeit
***** Running Emacs startup time measurement tests
** Report Configuration settings.
emacs --batch -L . -l "~/.emacs.d/init.el" -l pel-package.el -f pel-package-info
Loading /Users/roup/.emacs.d/emacs-customization.el (source)...
Loading pel_keys...
Loading /Users/roup/.emacs.d/recentf...
Cleaning up the recentf list...
Cleaning up the recentf list...done (0 removed)
PEL loaded, PEL keys binding in effect.
size of load-path           : 241 directories
Number of PEL user-options  : 250 (198 are active)
PEL activated elpa  packages: 167 ( 45 dependants, 5 imposed by restrictions)
PEL Activated utils files   :  25 (  0 dependants, 0 imposed by restrictions)

** Time mesaurement:
time -p emacs -nw -Q -e kill-emacs
real         0.13
user         0.02
sys          0.01
time -p emacs -nw -q -e kill-emacs
real         0.13
user         0.02
sys          0.01
time -p emacs -nw -e kill-emacs
real         1.66
user         0.98
sys          0.55
>

As a small (yet incomplete) experiment:

  • I created another directory (~/tmp/roup).
    • Inside that directory I created symlinks for all .el and .elc files of elpa sub-directories that do not have subdirectories (the 182 directories described above).
  • I placed that ~/tmp/roup directory at the front of Emacs load-path.
  • I left everything else unchanged.

Then I ran the same test a couple of times and I got something like this:

> make timeit
***** Running Emasc startup time measurement tests
** Report Configuration settings.
emacs --batch -L . -l "~/.emacs.d/init.el" -l pel-package.el -f pel-package-info
Loading /Users/roup/.emacs.d/emacs-customization.el (source)...
Loading pel_keys...
Loading /Users/roup/.emacs.d/recentf...
Cleaning up the recentf list...
Cleaning up the recentf list...done (0 removed)
PEL loaded, PEL keys binding in effect.
size of load-path           : 242 directories
Number of PEL user-options  : 250 (198 are active)
PEL activated elpa  packages: 167 ( 45 dependants, 5 imposed by restrictions)
PEL Activated utils files   :  25 (  0 dependants, 0 imposed by restrictions)

** Time mesaurement:
time -p emacs -nw -Q -e kill-emacs
real         0.13
user         0.02
sys          0.01
time -p emacs -nw -q -e kill-emacs
real         0.13
user         0.02
sys          0.01
time -p emacs -nw -e kill-emacs
real         1.50
user         0.92
sys          0.46
>

The time measurement is not very accurate as it is affected by other system operations, but I ran them several times and got similar results.

So just by providing a directory with symlinks to several of the files that are used during startup I was able to shave about 0.16 second from startup.
That's a 10% reduction of the startup time.

With Emacs 27.2, with or without package-quickstart, I get similar results (yet Emacs 27.2 is a little faster.)

By removing the 182 directories from the ~/.emacs.d/elpa directory and place the .el and .elc files into ~/.emacs.d/elpa-files then put this in front of the load-path (replacing the directory with symlinks) I reduced the time back to 0.77 secs:

** Time measurement:
time -p emacs -nw -Q -e kill-emacs
real         0.13
user         0.02
sys          0.01
time -p emacs -nw -q -e kill-emacs
real         0.14
user         0.03
sys          0.01
time -p emacs -nw -e kill-emacs
real         0.77
user         0.58
sys          0.08
>

The ~/.emacs.d/elpa-files holds 1612 files. This is a large number of files but the end result is that it reduces the startup time quite a bit.

Question: Would it not be possible to modify the design of Emacs package managing logic to store all the .el and .elc inside a same directory?

We would have to create some other files to remember the package version number that have been installed and their files. But that processing would only occur during a package install or removal, it would not affect Emacs startup time.

Is there something in package.el that makes this strategy difficult or impossible?

  • Currently the package inter-dependency checking breaks when package directories identifying the dependents and package detection fail.s But I assume this could be overcome if the installed packages were identified in some form of local db/list.
Improve this question
2
  • How often do you start an emacs session? For me, the answer to that question is very close to 0: I use the same session (actually two sessions) for days, sometimes weeks on end. Saving a few seconds every couple of days or every weeks just doesn't seem too important. And there are convenience reasons for not putting them all in the same directory: large directories are a pain. IMO of course.
    – NickD
    Jun 30 '21 at 22:46
  • I have various use cases (but that's beside the point:-) Some of my sessions last weeks with Emacs code changing during that and even crashing and recovering. Others last minutes. And they are from different environments, for various reasons. I know we can use various techniques to deal with "don't start a new Emacs process" :-). What I am after is understanding the package management mechanisms. The question was just to put a context to that, not to state that I don't want to use the other techniques that are often referred to.
    – PRouleau
    Jul 1 '21 at 1:02
2

Sure, that’s possible. However, note that you will somehow have to ensure that filenames from different packages remain distinct. Currently nothing prevents two unrelated packages from both having a foo.el file.

There are however two other techniques that can improve startup performance without needing to make radical changes to the package implementation.

The first is simply to avoid using require in your init files. If you’re going to use a package called foo, and you want to configure some of foo’s variables, you do not need to run (require 'foo) before doing so. Just set the variables to whatever values you want them to have. Emacs will note that these variables are not yet defined, but will still store your value for them. Later, the first time you call a function from the foo package, foo.el will be read in and the package will be fully useable.

The second is to configure and require all the packages you use frequently, then dump a new Emacs image. You can either use the old unexec dumper or the newer portable dumper; either way this ensures that Emacs need not run a complicated init file; most of your customizations are already available in the image. See the help for the functions dump-emacs and dump-emacs-portable for more information, as well as appendix E.1 Building Emacs of the Emacs manual (which is also available inside of Emacs; just run C-h i to open the Info viewer).

Improve this answer
5
  • Thanks for your answer. I already delay file loading, & leaving off the Emacs dump and daemon techniques for now. What interest me though is "currently nothing prevents two unrelated packages from having 2 identical file names". I have not seen it very often (aside from elpa.el being inside both lispy and ivy). So if best practice is to circumvent the single name space problem with prefix in file names, why is package.el not taking advantage of that to speed up Emacs. Only because of fear of a clash? A clash would affect the operation anyway.
    – PRouleau
    Jun 30 '21 at 21:58
  • A filename clash between packages would not break those packages. Of course they could break if they both define functions or variables with the same name, but having the same filename is ok. If you synthesize a new file name, you may run into file name length limits. Dumping is the far superior solution because it converts hundreds of individual elisp files into a single memory image. It eliminates not just searching through multiple directories to find the right file, but also the work of stating, reading, parsing, and executing them.
    – db48x
    Jun 30 '21 at 22:12
  • Let's keep Emacs dumping out of the equation, I understand its benefits. I'm trying to understand the loading of code from files with the same name: how does Emacs access function foo from file package-bar/abc.el and then access a different foo from file package-other/abc.el ? I understand that if I only have one of them in the system all is hanky dory. But when both are present, how can they both be accessible? Does Emacs package management code always use the full path of each directory when performing the loading (as can be specified in require as the FILENAME argument?
    – PRouleau
    Jun 30 '21 at 22:48
  • The should probably be a separate question, but I’ll try to give the short version here. If both files have a function called foo, then whichever one you loaded second will overwrite the one you loaded first; that second foo will redefine the function. If one has a function foo and the other has bar, then once you’ve loaded both then you will have two functions as expected. This is true no matter what directories the files are in, whether the files have the same base name or not, and whether you use load or require to load the file.
    – db48x
    Jul 1 '21 at 15:10
  • Thanks, that makes sense.
    – PRouleau
    Jul 1 '21 at 18:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.