My new init.el uses require to load various settings that I have organised into directories and files. This is an attempt to tame what was a very large and messy init.el and custom.el file. Also I ultimately hope to use org files to better document each setting.

I'm trying (and largely failing) to make this setup play nicely with customize.

My old init.el has grown over a few years and makes liberal use of setq.

But reading this: Advantages of setting variables with setq instead of custom.el?

And these: https://www.reddit.com/r/emacs/comments/b1jzno/need_help_making_my_initel_file_look_cleaner_its/

https://www.reddit.com/r/emacs/comments/9rrhy8/emacsers_with_beautiful_initel_files_what_about/

I decided to try replacing setq with custom-set-variables for custom variables.

The problem I have is that despite the fact that I've explicitly set variables using custom-set-variables, customize appears to take all my explicit settings and appends them to the end of my init.el or in a separate file if custom-file is set. This duplicates the settings needlessly, and create more, not less mess.

What's worse is that complicated expressions seems to get rendered incorrectly when trampled, the below:

(let* ((default-directory temporary-file-directory)
       (emacs-temp (expand-file-name current-user "emacs-saves")))
  (custom-set-variables '(backup-directory-alist (`(("." . ,emacs-temp))))
            '(auto-save-file-name-transforms (`(("." ,emacs-temp t))))))

Is trampled with custom.el containing an out of scope emacs-temp. No such issues are seen using setq:

(custom-set-variables
 ;; custom-set-variables was added by Custom.
 ;; If you edit it by hand, you could mess it up, so be careful.
 ;; Your init file should contain only one such instance.
 ;; If there is more than one, they won't work right.
 '(auto-save-file-name-transforms ((\` (("." (\, emacs-temp) t)))))
 '(backup-by-copying t)
 '(backup-directory-alist ((\` (("." \, emacs-temp))))))

It appears I can prevent customize writing anything out by using:

(setq custom-file null-device)

But this is problematic because customize settings made outside of my init.el and it's require statements, will then never be recorded. Apart from my own (very occasional) use of customize, other packages may try to write to the custom file - eg package-selected-packages.

The way I see it - customize should only be writing out settings that are not already explicitly set in init.el, etc. This will avoid trampling of any settings I've explicitly made.

Any other settings made within customize (or elsewhere) that would not trample anything explicitly set in init.el, could then be appended to init.el, or directed elsewhere by custom-file.

The closest behaviour to this seems to be the use of setq not custom-set-variables which seems to be ignored by customize. However I take the points that use of defcustom keywords means that setq use can be problematic.

My question is - how do I correctly and safely initialize custom values in my initialization script(s), such that they are not then trampled by customize saving these values back down to a file, and without preventing other customize values from being saved to file?

Will use-package help me? Can I make setq use safe without interleaving setup before and after initialization?

EDIT

Responding to some feedback, I'm going to walk through a few simple cases step by step.

As you'll see below following through a simple example as suggested by @Drew yields the correct result. However a more complex example using require to pull in programmatically set customizable variables reproduces the issue.

Attempt 1: Simplest Example - This works as expected

1. Take a fresh install of emacs (for this I used v25.2.2)
2. Inside the empty ~/.emacs.d place an init.el file containing a single line (setq backup-by-copying t). ~/.emacs.d$ echo '(setq backup-by-copying t)' > init.el. This represents a value that I do not want to set directly in customize as I want to organize my own setup into more readable sections over several files/directories - I've picked backup-by-copying at arbitrarily.
3. I then run emacs and see the following in customize - Backup By Copying, [ State ]: CHANGED outside Customize..
4. So far no surprises. Now let's set a completely unrelated item directly in customize. The logic here, is that either via automated other packages or directly by me, it's possible that something is required to be recorded directly in customize appended to init.el (by default). So in the customize UI I search for and Toggle Create Lockfiles.
5. This gives me [ State ]: EDITED, shown value does not take effect until you set or save it.
6. To save this (and hopefully only this) setting to the end of init.el I'm clicking on [ Apply and Save ]
7. emacs confirms Wrote /home/foo/.emacs.d/init.el
8. The contents of my init.el is now:

(setq backup-by-copying t)
(custom-set-variables
 ;; custom-set-variables was added by Custom.
 ;; If you edit it by hand, you could mess it up, so be careful.
 ;; Your init file should contain only one such instance.
 ;; If there is more than one, they won't work right.
 '(create-lockfiles nil))
(custom-set-faces
 ;; custom-set-faces was added by Custom.
 ;; If you edit it by hand, you could mess it up, so be careful.
 ;; Your init file should contain only one such instance.
 ;; If there is more than one, they won't work right.
 )

OK - so far everything is as expected, now I'll repeat, overwriting the original init.el, replacing setq with custom-set-variables in point 2:

1. As above
2. ~/.emacs.d$ echo "(custom-set-variables '(backup-by-copying t))" > init.el
3. In customize I now see Backup By Copying set to [ State ]: SAVED and set.
4. Same as above
5. Same as above
6. Same as above
7. Same as above
8. The contents of my init.el is now

(custom-set-variables
 ;; custom-set-variables was added by Custom.
 ;; If you edit it by hand, you could mess it up, so be careful.
 ;; Your init file should contain only one such instance.
 ;; If there is more than one, they won't work right.
 '(backup-by-copying t)
 '(create-lockfiles nil))
(custom-set-faces
 ;; custom-set-faces was added by Custom.
 ;; If you edit it by hand, you could mess it up, so be careful.
 ;; Your init file should contain only one such instance.
 ;; If there is more than one, they won't work right.
 )

Nothing wrong with this - '(create-lockfiles nil) has been pulled under the control of customize. There is a single, correct, copy of the setting!

So now I will try to isolate the case I previous had seen.

Attempt 2: Using require Example - WIP

Repeating what was done in Attempt 1 above, but with a sightly more complex init.el structure.

Create a subdirectory under .emacs.d called settings, this is going to contain a single lisp file file-settings.el which we will require from init.el.

1. echo "(add-to-list 'load-path (expand-file-name \"settings\" user-emacs-directory))" > init.el followed by echo "(require 'file-settings)" >> init.el
2. echo "(setq backup-by-copying t)" > settings/file-settings.el followed by echo "(provide 'file-settings)" >> settings/file-settings.el
3. Same as setq for Attempt 1 - [ State ]: CHANGED outside Customize.
4. Same as Attempt 1
5. Same as Attempt 1
6. Same as Attempt 1
7. Same as Attempt 1
8. The contents of my init.el is now:

(add-to-list 'load-path (expand-file-name "settings" user-emacs-directory))
(require 'file-settings)
(custom-set-variables
 ;; custom-set-variables was added by Custom.
 ;; If you edit it by hand, you could mess it up, so be careful.
 ;; Your init file should contain only one such instance.
 ;; If there is more than one, they won't work right.
 '(create-lockfiles nil))
(custom-set-faces
 ;; custom-set-faces was added by Custom.
 ;; If you edit it by hand, you could mess it up, so be careful.
 ;; Your init file should contain only one such instance.
 ;; If there is more than one, they won't work right.
 )

So far all good. Now, again, let's swap-out setq for custom-set-variables

1. As above
2. echo "(custom-set-variables '(backup-by-copying t))" > settings/file-settings.el followed by echo "(provide 'file-settings)" >> settings/file-settings.el
3. Same as Attempt 1 - In customize I now see Backup By Copying set to [ State ]: SAVED and set.
4. Same as above
5. Same as above
6. Same as above
7. Same as above
8. The contents of my init.el is now

(add-to-list 'load-path (expand-file-name "settings" user-emacs-directory))
(require 'file-settings)
(custom-set-variables
 ;; custom-set-variables was added by Custom.
 ;; If you edit it by hand, you could mess it up, so be careful.
 ;; Your init file should contain only one such instance.
 ;; If there is more than one, they won't work right.
 '(backup-by-copying t)
 '(create-lockfiles nil))
(custom-set-faces
 ;; custom-set-faces was added by Custom.
 ;; If you edit it by hand, you could mess it up, so be careful.
 ;; Your init file should contain only one such instance.
 ;; If there is more than one, they won't work right.
 )

BUT the contents of settings/file-settings.el is still:

(custom-set-variables '(backup-by-copying t))
(provide 'file-settings)

So the setting of backup-by-copying has been duplicated?

Is it possible to use require (or similar) to organize my config into a modular structure of directories, without running into this issue?