High CPU/memory usage and abnormally large savehist file?

Question

I'm starting to use Spacemacs and occasionally I get stuck with Emacs beginning to require infinite amount of CPU and memory. Inspecting my .emacs.d/.cache folder reveals that the file savehist is 1.16 GB large... Deleting this file made Emacs work again.

Googling returned no result for me. Is there any precedent of such an abnormality, and how might one try to debug and fix it.

Answer 1

I would guess that's a result of one specific persistent variable getting enormous (it just seems less likely that you would have multiple instances at the same time), but if you've deleted the file you can't see which variable it was.

You can always rename the file rather than deleting it, so that you can make Emacs happy again, but still analyse the data.

If you recreate this situation, and the file is really massive, opening it in Emacs to check might not be a great idea. I would use grep to find out the offset for each variable, and just look for any really big jumps:

$ grep -E -b -o '^\(setq [^ ]+' ~/.emacs.d/.cache/savehist

(Or whatever your savehist-file path is -- the standard filename is ~/.emacs.d/history but I presume ~/.emacs.d/.cache/savehist is what Spacemacs configures.)

---

Edit: The results of this from the comments below were:

116:(setq savehist-minibuffer-history-variables 
350:(setq regexp-search-ring 
566:(setq magit-read-rev-history 
637:(setq read-expression-history 
694:(setq command-history 
19045:(setq helm-grep-history 
20687:(setq minibuffer-history 
75425:(setq file-name-history 
83278:(setq ido-file-history 
83377:(setq evil-ex-history 
111288:(setq kill-ring 
1162765006:(setq mark-ring 
1162765028:(setq search-ring 
1162765052:(setq regexp-search-ring 
1162765268:(setq extended-command-history

From this we can see that kill-ring is the biggest culprit by some orders of magnitude!

Edit 2: Strip text properties in savehist almost certainly accounts for the excessive size of the kill-ring entry. The remainder of this answer is still useful information, but I recommend also looking at that other Q&A.

---

It's possible that you need to fix a bug somewhere -- whatever is causing that value to become so huge. Even if you're not writing a value to the history file, Emacs may still be burning a lot of memory on that data (especially if you rarely quit Emacs, and it's accumulating rapidly).

If that's not an issue, though, it might be sufficient for Emacs to simply not persist the variable(s) in question -- if they're not persistent, the size won't be growing continually from session to session.

If you just don't want it written to the history file, and the offending variable is a member of savehist-additional-variables, then it's probably your own config which is responsible for having added it, so you should find and remedy that.

Otherwise it's most likely a minibuffer history variable which savehist is adding automatically, in which case you could add it to savehist-ignored-variables to prevent this from occuring.

Alternatively (and preferably) for variables which are explicitly used as histories, you should be able to alleviate this via the history-length variable, which means you can still maintain the most recent history for variables of interest. Note in particular that you can specify this on a per-variable basis:

This variable only affects history lists that don't specify their own
maximum lengths.  Setting the `history-length' property of a history
variable overrides this default.

e.g.:

(setq history-length 100)
(put 'minibuffer-history 'history-length 50)
(put 'evil-ex-history 'history-length 50)
(put 'kill-ring 'history-length 25)

n.b. If you have set the value of history-length to t, that disables truncation, and the affected histories will be allowed to grow without restraints! (of course an integer value which is simply very very large can be just as problematic.)

Note that setting a (sensible integer) history-length value is beneficial for Emacs' memory usage as well as the history file size, because the list truncation happens as history items are added, and so the in-memory variable never exceeds its maximum length + 1.

Also note that kill-ring is not a history variable, and so it is unaffected by history-length. It does however have its own analogous constraint in the kill-ring-max variable, so you should definitely check your setting for that.

You may like to make kill-ring an ignored variable regardless, unless you particularly want that to persist between sessions. (I don't think that it's persistent by default, mind, so you should maybe check why that's happening?)

Finally, you can utilise savehist-save-hook to run your own custom functions before the variables are written to the history file, so you can do pretty much anything you want at that point. For example if you wished to maintain a larger history-length while Emacs is running, but only write a smaller sub-set of that data to the history file when Emacs exits, then you could write a custom function to cut those lists down to size.

Answer 2

Phils' answer is great and put me on the right track to solve my similar issue.

However, I found that directly locating long lines in the history file gave more specific results.

cat $your_hist_file_backup | \
    awk '(length($0) > 1000) {printf (" - Line %d is %d characters long\n", NR, length($0) );}'

and then identify preceding setq line. I'm using cut to truncate long lines.

cat $your_hist_file_backup | \
    sed -n -e '/^(setq/ p' -e "${your_culprit_line_num}q" | \
    cut -c 1-80

In my case, it definitely established that kill-ring was to blame.

Check spacemacs-defaults/init-savehist() in file:layers/+spacemacs/spacemacs-defaults/packages.el