init.el and trampling of custom-set-variables

Question

My new init.el uses require to load various settings that I have organised into directories and files. This is an attempt to tame what was a very large and messy init.el and custom.el file. Also I ultimately hope to use org files to better document each setting.

I'm trying (and largely failing) to make this setup play nicely with customize.

My old init.el has grown over a few years and makes liberal use of setq.

But reading this: Advantages of setting variables with setq instead of custom.el?

And these: https://www.reddit.com/r/emacs/comments/b1jzno/need_help_making_my_initel_file_look_cleaner_its/

https://www.reddit.com/r/emacs/comments/9rrhy8/emacsers_with_beautiful_initel_files_what_about/

I decided to try replacing setq with custom-set-variables for custom variables.

The problem I have is that despite the fact that I've explicitly set variables using custom-set-variables, customize appears to take all my explicit settings and appends them to the end of my init.el or in a separate file if custom-file is set. This duplicates the settings needlessly, and create more, not less mess.

What's worse is that complicated expressions seems to get rendered incorrectly when trampled, the below:

(let* ((default-directory temporary-file-directory)
       (emacs-temp (expand-file-name current-user "emacs-saves")))
  (custom-set-variables '(backup-directory-alist (`(("." . ,emacs-temp))))
            '(auto-save-file-name-transforms (`(("." ,emacs-temp t))))))

Is trampled with custom.el containing an out of scope emacs-temp. No such issues are seen using setq:

(custom-set-variables
 ;; custom-set-variables was added by Custom.
 ;; If you edit it by hand, you could mess it up, so be careful.
 ;; Your init file should contain only one such instance.
 ;; If there is more than one, they won't work right.
 '(auto-save-file-name-transforms ((\` (("." (\, emacs-temp) t)))))
 '(backup-by-copying t)
 '(backup-directory-alist ((\` (("." \, emacs-temp))))))

It appears I can prevent customize writing anything out by using:

(setq custom-file null-device)

But this is problematic because customize settings made outside of my init.el and it's require statements, will then never be recorded. Apart from my own (very occasional) use of customize, other packages may try to write to the custom file - eg package-selected-packages.

The way I see it - customize should only be writing out settings that are not already explicitly set in init.el, etc. This will avoid trampling of any settings I've explicitly made.

Any other settings made within customize (or elsewhere) that would not trample anything explicitly set in init.el, could then be appended to init.el, or directed elsewhere by custom-file.

The closest behaviour to this seems to be the use of setq not custom-set-variables which seems to be ignored by customize. However I take the points that use of defcustom keywords means that setq use can be problematic.

My question is - how do I correctly and safely initialize custom values in my initialization script(s), such that they are not then trampled by customize saving these values back down to a file, and without preventing other customize values from being saved to file?

Will use-package help me? Can I make setq use safe without interleaving setup before and after initialization?

EDIT

Responding to some feedback, I'm going to walk through a few simple cases step by step.

As you'll see below following through a simple example as suggested by @Drew yields the correct result. However a more complex example using require to pull in programmatically set customizable variables reproduces the issue.

Attempt 1: Simplest Example - This works as expected

1. Take a fresh install of emacs (for this I used v25.2.2)
2. Inside the empty ~/.emacs.d place an init.el file containing a single line (setq backup-by-copying t). ~/.emacs.d$ echo '(setq backup-by-copying t)' > init.el. This represents a value that I do not want to set directly in customize as I want to organize my own setup into more readable sections over several files/directories - I've picked backup-by-copying at arbitrarily.
3. I then run emacs and see the following in customize - Backup By Copying, [ State ]: CHANGED outside Customize..
4. So far no surprises. Now let's set a completely unrelated item directly in customize. The logic here, is that either via automated other packages or directly by me, it's possible that something is required to be recorded directly in customize appended to init.el (by default). So in the customize UI I search for and Toggle Create Lockfiles.
5. This gives me [ State ]: EDITED, shown value does not take effect until you set or save it.
6. To save this (and hopefully only this) setting to the end of init.el I'm clicking on [ Apply and Save ]
7. emacs confirms Wrote /home/foo/.emacs.d/init.el
8. The contents of my init.el is now:

(setq backup-by-copying t)
(custom-set-variables
 ;; custom-set-variables was added by Custom.
 ;; If you edit it by hand, you could mess it up, so be careful.
 ;; Your init file should contain only one such instance.
 ;; If there is more than one, they won't work right.
 '(create-lockfiles nil))
(custom-set-faces
 ;; custom-set-faces was added by Custom.
 ;; If you edit it by hand, you could mess it up, so be careful.
 ;; Your init file should contain only one such instance.
 ;; If there is more than one, they won't work right.
 )

OK - so far everything is as expected, now I'll repeat, overwriting the original init.el, replacing setq with custom-set-variables in point 2:

1. As above
2. ~/.emacs.d$ echo "(custom-set-variables '(backup-by-copying t))" > init.el
3. In customize I now see Backup By Copying set to [ State ]: SAVED and set.
4. Same as above
5. Same as above
6. Same as above
7. Same as above
8. The contents of my init.el is now

(custom-set-variables
 ;; custom-set-variables was added by Custom.
 ;; If you edit it by hand, you could mess it up, so be careful.
 ;; Your init file should contain only one such instance.
 ;; If there is more than one, they won't work right.
 '(backup-by-copying t)
 '(create-lockfiles nil))
(custom-set-faces
 ;; custom-set-faces was added by Custom.
 ;; If you edit it by hand, you could mess it up, so be careful.
 ;; Your init file should contain only one such instance.
 ;; If there is more than one, they won't work right.
 )

Nothing wrong with this - '(create-lockfiles nil) has been pulled under the control of customize. There is a single, correct, copy of the setting!

So now I will try to isolate the case I previous had seen.

Attempt 2: Using require Example - Doesn't work as I had expected

Repeating what was done in Attempt 1 above, but with a sightly more complex init.el structure.

Create a subdirectory under .emacs.d called settings, this is going to contain a single lisp file file-settings.el which we will require from init.el.

1. echo "(add-to-list 'load-path (expand-file-name \"settings\" user-emacs-directory))" > init.el followed by echo "(require 'file-settings)" >> init.el
2. echo "(setq backup-by-copying t)" > settings/file-settings.el followed by echo "(provide 'file-settings)" >> settings/file-settings.el
3. Same as setq for Attempt 1 - [ State ]: CHANGED outside Customize.
4. Same as Attempt 1
5. Same as Attempt 1
6. Same as Attempt 1
7. Same as Attempt 1
8. The contents of my init.el is now:

(add-to-list 'load-path (expand-file-name "settings" user-emacs-directory))
(require 'file-settings)
(custom-set-variables
 ;; custom-set-variables was added by Custom.
 ;; If you edit it by hand, you could mess it up, so be careful.
 ;; Your init file should contain only one such instance.
 ;; If there is more than one, they won't work right.
 '(create-lockfiles nil))
(custom-set-faces
 ;; custom-set-faces was added by Custom.
 ;; If you edit it by hand, you could mess it up, so be careful.
 ;; Your init file should contain only one such instance.
 ;; If there is more than one, they won't work right.
 )

So far all good. Now, again, let's swap-out setq for custom-set-variables

1. As above
2. echo "(custom-set-variables '(backup-by-copying t))" > settings/file-settings.el followed by echo "(provide 'file-settings)" >> settings/file-settings.el
3. Same as Attempt 1 - In customize I now see Backup By Copying set to [ State ]: SAVED and set.
4. Same as above
5. Same as above
6. Same as above
7. Same as above
8. The contents of my init.el is now

(add-to-list 'load-path (expand-file-name "settings" user-emacs-directory))
(require 'file-settings)
(custom-set-variables
 ;; custom-set-variables was added by Custom.
 ;; If you edit it by hand, you could mess it up, so be careful.
 ;; Your init file should contain only one such instance.
 ;; If there is more than one, they won't work right.
 '(backup-by-copying t)
 '(create-lockfiles nil))
(custom-set-faces
 ;; custom-set-faces was added by Custom.
 ;; If you edit it by hand, you could mess it up, so be careful.
 ;; Your init file should contain only one such instance.
 ;; If there is more than one, they won't work right.
 )

BUT the contents of settings/file-settings.el is still:

(custom-set-variables '(backup-by-copying t))
(provide 'file-settings)

So the setting of backup-by-copying has been duplicated?

Is it possible to use require (or similar) to organize my config into a modular structure of directories, without running into this issue?

Answer 1

It sounds like you have done both of the following for some options, i.e., you've done them both for at least one option:

1. Used Customize to set and save the option.
2. Tried to call customize functions to set the option in your init file.

If so, it's you who introduced the redundancy. For a given option, choose one or the other. For example, if you want to define backup-directory-alist dynamically, making it dependent on your local variable emacs-temp, then remove its setting from your custom-file.

Use Customize to set only options whose values are not variable, i.e., they don't depend on anything else or any conditional handling. That's what custom-file is for. The settings you provide for it are not sexps that get evaluated when custom-file is loaded.

Put any fancy handling of other options in your init file (or a Lisp file that it loads).

And yes, your fancy handling of, say, backup-directory-alist, could be done using setq, but the effect would be the same.

(The effect won't be the same in general, because of the possibility of :set etc. defcustom parameters. But the fact of changing the value after it was first set via your custom-file is the same.)

The problem is not that Customize "trample[s]" on such a value "by saving these values back down to a file". The problem is that you saved such values using Customize in the first place.

---

In your comments here you say that if you change an option, say foo, outside customize (doesn't matter how), and then you use the Customize UI to set and save some other option, say, bar, then both options foo and bar are written to your custom-file.

That doesn't happen if you just use Customize to set and save bar. You must be telling Customize to save all changed, unsaved options. Please update your recipe (in the question, not in comments) to clarify just what you are doing, and be sure to start Emacs using emacs -Q (no init file), to eliminate distractions.

If you start Emacs with emacs -Q and you do what I described (set option foo without using Customize and set and save (only) option bar using Customize, then only bar should be written to your custom-file.

---

See also Emacs-Wiki page Customizing and Saving for more info about Customize.

Answer 2

Is it possible to use require (or similar) to organize my config into a modular structure of directories, without running into this issue?

Yes. Don't add custom-set-variables to multiple places in your init. There should be exactly one instance of custom-set-variables. By default, Emacs creates this in your init.el file. If it's already there, you cannot use that form anywhere else without causing problems.

For your intended approach, you should be using customize-set-variable. Similar spelling, but important difference!