I can think of three methods (in the following, for definiteness, I assume you use the recursive method to construct org-agenda-files in your init file; the changes required for the list method should be obvious). In increasing order of complexity:
Interactively add every new agenda file
If you create a new file, add content and at some point you decide that the file needs to contribute to your agenda, you can add the file to org-agenda-files using C-c [ (which is bound to org-agenda-file-to-front). If you then create a new agenda, or refresh an existing one with g, the file will contribute to the agenda during the current emacs session. If you save the file in a directory that is used to set org-agenda-files during initialization (e.g. with your recursive setting above, you save it in the file ~/org/my_subdir/my_file.org, i.e. in a subdirectory of ~/org with a file name that has a .org suffix), the initialization code will include that file in org-agenda-files every time afterwards, so it will contribute to your agenda.
Re-evaluate org-agenda-files on demand
Instead of interactively adding files, you can create a bunch of them, save them under the ~/org directory, in files whose name have the .org suffix, and then after you are done and before you recreate/refresh the agenda, re-evaluate org-agenda-files. That can be done in various ways: switch to the *scratch* buffer, type the same sexpr that you used in the init file:
(setq org-agenda-files (directory-files-recursively "~/org/" "\\.org$"))
and then press C-j: the *scratch* buffer should be in lisp-interaction-mode where C-j is bound to eval-print-last-sexp. That will give a new value to org-agenda-files and print it out so you can verify it.
Recreating/refreshing the agenda will use the new value of org-agenda-files so the newly added files will contribute to it.
Slight variations of this are:
re-evaluate by opening your init file, finding the expression that initializes org-agenda-files in the first place, position the cursor after the closing paren and pressing C-x C-e: the init file should be in emacs-lisp-mode (like all .el files) where C-x C-e is bound to eval-last-sexp. In this case the value is printed in the Echo Area (and will be probably elided because it will be too long), but also in the Messages buffer where you can go and verify it.
you can write a command to do the re-evaluation and bind it to a key:
(defun ndk/refresh-org-agenda-files ()
(interactive)
(setq org-agenda-files (directory-files-recursively "~/org/" "\\.org$")))
;; I'm assuming that `C-c z` - one of the keys that is reserved for
;; end-user use - is undefined. You can bind the function above to it
;; in the global map, so that it is always available - unless a mode redefines it...
(global-set-key (kbd "C-c z") #'ndk/refresh-org-agenda-files)
;; ... or you add the binding to the org-mode-map,
;; so that it is only available in Org mode buffers.
(define-key org-mode-map (kbd "C-c z") #'ndk/refresh-org-agenda-files)
Now pressing C-c z will execute the command and re-evaluate org-agenda-files based on the new contents.
Have emacs monitor the directory automatically
On Linux (and some other OSes), you can use inotify to monitor the directory: when some event happens, inotify can tell you about it and you can call e.g. the above function to refresh org-agenda-files when a change happens. The trick is to have Emacs do the listening and the refreshing automatically.
The idea is to create an asynchronous sub-process that uses inotify to monitor changes to the directory and, when a change is detected, to run the function above to re-evaluate org-agenda-files using the newly-changed contents.
Here is a bash command line that monitors a directory recursively:
inotifywait -e create -e delete -m -r ~/src/inotifywait/watched
Then when a file (or directory) is created or deleted in the watched directory or one of its subdirectories, the command prints out a message. E.g. here is what is printed out when I create and then delete a file z in the (already existing) watched/e subdirectory and then create and remove a subdirectory f under the watched directory:
$ inotifywait -e create -e delete -m -r ~/src/inotifywait/watched
Setting up watches. Beware: since -r was given, this may take a while!
Watches established.
/home/me/src/inotifywait/watched/e/ CREATE z
/home/me/src/inotifywait/watched/e/ DELETE z
/home/me/src/inotifywait/watched/ CREATE,ISDIR f
/home/me/src/inotifywait/watched/ DELETE,ISDIR f
...
The command runs continuously (because of the -m monitoring option), watches every subdirectory (-r) of the given directory, printing out messages to its stdout whenever a file or a directory is created or deleted under it. You should read the man page inotifywait(1) for all the gory details. One caveat: doing -r on a large directory hierarchy can have adverse effects on your system performance: inotify has to set up watches on every subdirectory (and depending on the events, perhaps on every file); doing it on a handful of directories with a few tens or hundreds of files should be no problem on a modern machine with lots of memory, but YMMV, so "Caveat Emptor"!
Now to have emacs do all this: run the above command as an asynchronous subprocess, watch its output and whenever it sees something interesting in that output, call the above function to re-evaluate org-agenda-files.
To start the process, you need to add something like the following bit of code to your init file (or evaluate it by hand if you want to test things out: C-x C-e after the closing paren in an emacs-lisp buffer will do it):
(setq ndk/inotify-org-agenda-process
(make-process
:name "ndk-inotify-org-agenda"
:command `("inotifywait"
"-e" "create"
"-e" "delete"
"-m" "-r" ,(expand-file-name "~/org"))
:buffer (get-buffer-create "*ndk/inotify-org-agenda-process-buffer*")
:connection-type 'pipe
:stderr (get-buffer-create "*ndk/inotify-org-agenda-process-stderr*")))
This uses the low-level make-process to create a process that runs the given command. It arranges so that the stdout of the command is inserted into the process buffer *ndk/inotify-org-agenda-process-buffer* and its stderr is inserted into the *ndk/inotify-org-agenda-process-stderr* buffer.
If you evaluate this code and visit those two buffers, you should be able to see output from the command whenever you create or delete a file in the ~/org directory or its subdirectories (it will also produce output when you create or delete a subdirectory, which we'll have to filter out, since the ultimate aim is to detect creation or deletion of Org mode files).
We now have to add a process filter to the above, so that we can examine the output and act accordingly. Read the "Processes" chapter in the Emacs Lisp Manual with C-h i g(elisp)Processes. In particular, the section Receiving Output from Processes describes process filters.
To add a process filter, we define a filter function and change the call to make-process above to specify the filter:
(setq ndk/inotify-org-agenda-process
(make-process
:name "ndk-inotify-org-agenda"
:command `("inotifywait"
"-e" "create"
"-e" "delete"
"-m" "-r" ,(expand-file-name "~/org"))
:buffer (get-buffer-create "*ndk/inotify-org-agenda-process-buffer*")
:connection-type 'pipe
:filter #'ndk/inotify-org-agenda-process-filter
:stderr (get-buffer-create "*ndk/inotify-org-agenda-process-stderr*")))
(defun ndk/inotify-org-agenda-process-filter (proc txt)
(when (buffer-live-p (process-buffer proc))
(display-buffer (process-buffer proc))
(with-current-buffer (process-buffer proc)
(let ((begin (marker-position (process-mark proc))))
;; Insert the text, advancing the process marker.
(goto-char (process-mark proc))
(insert-before-markers txt)
(set-marker (process-mark proc) (point))
(goto-char (process-mark proc))
;; the above is pretty generic: the reason we need a custom
;; filter function is so that we can test whether the process
;; output indicates that we should refresh
;; `org-agenda-files'. We remember where the previous output
;; had ended in `begin'; so the new output is the region
;; between `begin' and the current `point'. We pass that
;; information to the checker and if it returns non-nil, we
;; refresh `org-agenda-files'.
(if (ndk/update-org-agenda-file-list? begin (point))
(ndk/refresh-org-agenda-files))))))
If you don't specify a process filter, a default one is used that adds the output of the process to the process buffer. Here we define a custom filter that does the same thing but adds a few wrinkles. Filter functions take two arguments: the process and the string that the process has output since the last time the filter was called. We remember where the previous output ended, i.e. where our new output should go: that's the process mark. We make sure to go to that marker before inserting the new output, pushing any markers forward as we do so. We then set the process mark to the end of the new text and make sure that we go there (this should be a no-op, but a little paranoia never hurt). That's fairly generic: just about any filter would do a similar thing (but since this is the first filter I've ever written, if there are any problems, this is where I would look first: I wonder if this simple filter is too simple, although I have not seen any problems with it - yet). Now for the particular stuff: given the beginning and end of the insertion, we can always grab it and pass it to the checker to see if there are new files named 'foo.org' that were created; if so, we call ndk/refresh-org-agenda-files to go get the new list (the latter function was defined above).
Although the checker should probably check e.g. that what was created or deleted was actually an Org file (and not e.g. a directory or a non-Org file), that's a bit involved and would make this long answer even longer. So I just provide a dummy implementation here which always returns t (i.e. whatever inotify reports, we always refresh org-agenda-files), and leave the "real" implementation to the interested reader: it's an exercise in parsing strings but does not have much to do with processes. Here's the dummy implementation:
(defun ndk/update-org-agenda-file-list? (begin end)
" Check whether the contents of the buffer (assumed to contain the
output of the inotify process) in the region between `begin' and
`end', indicate that we need to refresh `org-agenda-files'."
;; This is a simple example that always returns `t' (i.e.*anything*
;; added to or removed from the directory will trigger a refresh of
;; `org-agenda-files').
;;
;; But you could add some filtering, e.g. grab the contents of the
;; buffer using `buffer-substring' and parse it to figure out if the
;; file(s) that were added or removed should affect the agenda list
;; (e.g. adding or removing a subdirectory or a file that does not
;; end in `.org' should not trigger a a refresh). There may be some
;; addditional complications as well: it is not necessarily true
;; that the output of the command is spat out to the filter function
;; in one piece, so it may be necessary to deal with partial
;; lines. The simple implementation sidesteps these complications
;; but they must be kept in mind in general.
t)
The comment describes some details of a possible implementation and some caveats that might arise (although I have not seen them in practice).
If you want to kill the process and the associated buffers, here's a (rough-and-ready) function to do it:
(defun ndk/inotify-org-agenda-process-cleanup ()
(interactive)
(delete-process ndk/inotify-org-agenda-process)
(let ((kill-buffer-query-functions nil))
(kill-buffer "*ndk/inotify-org-agenda-process-buffer*")
(kill-buffer "*ndk/inotify-org-agenda-process-stderr*")))
Invoke with M-x ndk/inotify-org-agenda-process-cleanup. It kills the process and the two buffers, no questions asked, but it leaves a lot to be desired, in particular error handling.
C-c [(bound toorg-agenda-file-to-front). That will not persist to the next session, but if you save the file in one of the directories above, it will be added to the list at iniitialization. Maybe that's good enough?(setq org-agenda-files ...)again after you added new files into those directories: open your initialization file, place the cursor after the closing paren and doC-x C-e(bound toeval-last-sexp). That will updateorg-agenda-filesand the next time you create an agenda, the new files will contribute. It will also work if you refresh an existing agenda (withg). You can define a function that does the re-evaluation and bind it to a key: just hitting the key would do it. Or you can create a timer to call the function at some interval.