1

I am new to org and agenda, so this might be a newbie question...

I like to keep my daily notes in date-stamped files, and using the code snips below I am able to have emacs load them on startup. But if I add a new .org file to ~/org/daily-notes after emacs is running, I have to completely restart to make it show up on the agenda :(

In my startup .el code I have tried the following two solutions:

(after! org
(setq org-agenda-files (directory-files-recursively "~/org/" "\\.org$")))

and

(after! org
(setq org-agenda-files '("~/org/daily-notes/" "~/org/projects")))

both of these successfully load .org files, the first one searches sub-directories.

I don't expect emacs to magically watch the directory, but there has to be some command I can run to re-consume those folders...

Thank you for your help :)

3
  • You can dynamically add the current Org mode file to the front of the list of agenda files with C-c [ (bound to org-agenda-file-to-front). That will not persist to the next session, but if you save the file in one of the directories above, it will be added to the list at iniitialization. Maybe that's good enough?
    – NickD
    Jul 29 at 0:07
  • @NickD That definitely helps! It is at least a better work-around than restarting emacs... I would still love to find a more automated solution, but this will keep workflow (and my sanity) for the time being Jul 29 at 0:12
  • You could also evaluate the (setq org-agenda-files ...) again after you added new files into those directories: open your initialization file, place the cursor after the closing paren and do C-x C-e (bound to eval-last-sexp). That will update org-agenda-files and the next time you create an agenda, the new files will contribute. It will also work if you refresh an existing agenda (with g). You can define a function that does the re-evaluation and bind it to a key: just hitting the key would do it. Or you can create a timer to call the function at some interval.
    – NickD
    Jul 29 at 3:45
2

I can think of three methods (in the following, for definiteness, I assume you use the recursive method to construct org-agenda-files in your init file; the changes required for the list method should be obvious). In increasing order of complexity:

Interactively add every new agenda file

If you create a new file, add content and at some point you decide that the file needs to contribute to your agenda, you can add the file to org-agenda-files using C-c [ (which is bound to org-agenda-file-to-front). If you then create a new agenda, or refresh an existing one with g, the file will contribute to the agenda during the current emacs session. If you save the file in a directory that is used to set org-agenda-files during initialization (e.g. with your recursive setting above, you save it in the file ~/org/my_subdir/my_file.org, i.e. in a subdirectory of ~/org with a file name that has a .org suffix), the initialization code will include that file in org-agenda-files every time afterwards, so it will contribute to your agenda.

Re-evaluate org-agenda-files on demand

Instead of interactively adding files, you can create a bunch of them, save them under the ~/org directory, in files whose name have the .org suffix, and then after you are done and before you recreate/refresh the agenda, re-evaluate org-agenda-files. That can be done in various ways: switch to the *scratch* buffer, type the same sexpr that you used in the init file:

(setq org-agenda-files (directory-files-recursively "~/org/" "\\.org$"))

and then press C-j: the *scratch* buffer should be in lisp-interaction-mode where C-j is bound to eval-print-last-sexp. That will give a new value to org-agenda-files and print it out so you can verify it.

Recreating/refreshing the agenda will use the new value of org-agenda-files so the newly added files will contribute to it.

Slight variations of this are:

  • re-evaluate by opening your init file, finding the expression that initializes org-agenda-files in the first place, position the cursor after the closing paren and pressing C-x C-e: the init file should be in emacs-lisp-mode (like all .el files) where C-x C-e is bound to eval-last-sexp. In this case the value is printed in the Echo Area (and will be probably elided because it will be too long), but also in the Messages buffer where you can go and verify it.

  • you can write a command to do the re-evaluation and bind it to a key:

(defun ndk/refresh-org-agenda-files ()
   (interactive)
   (setq org-agenda-files (directory-files-recursively "~/org/" "\\.org$")))

;; I'm assuming that `C-c z` - one of the keys that is reserved for
;; end-user use - is undefined. You can bind the function above to it
;; in the global map, so that it is always available - unless a mode redefines it...
(global-set-key (kbd "C-c z") #'ndk/refresh-org-agenda-files)

;; ... or you add the binding to the org-mode-map,
;; so that it is only available in Org mode buffers.
(define-key org-mode-map (kbd "C-c z") #'ndk/refresh-org-agenda-files)

Now pressing C-c z will execute the command and re-evaluate org-agenda-files based on the new contents.

Have emacs monitor the directory automatically

On Linux (and some other OSes), you can use inotify to monitor the directory: when some event happens, inotify can tell you about it and you can call e.g. the above function to refresh org-agenda-files when a change happens. The trick is to have Emacs do the listening and the refreshing automatically.

The idea is to create an asynchronous sub-process that uses inotify to monitor changes to the directory and, when a change is detected, to run the function above to re-evaluate org-agenda-files using the newly-changed contents.

Here is a bash command line that monitors a directory recursively:

inotifywait -e create -e delete -m -r ~/src/inotifywait/watched

Then when a file (or directory) is created or deleted in the watched directory or one of its subdirectories, the command prints out a message. E.g. here is what is printed out when I create and then delete a file z in the (already existing) watched/e subdirectory and then create and remove a subdirectory f under the watched directory:

$ inotifywait -e create -e delete -m -r ~/src/inotifywait/watched
Setting up watches.  Beware: since -r was given, this may take a while!
Watches established.
/home/me/src/inotifywait/watched/e/ CREATE z
/home/me/src/inotifywait/watched/e/ DELETE z
/home/me/src/inotifywait/watched/ CREATE,ISDIR f
/home/me/src/inotifywait/watched/ DELETE,ISDIR f
...

The command runs continuously (because of the -m monitoring option), watches every subdirectory (-r) of the given directory, printing out messages to its stdout whenever a file or a directory is created or deleted under it. You should read the man page inotifywait(1) for all the gory details. One caveat: doing -r on a large directory hierarchy can have adverse effects on your system performance: inotify has to set up watches on every subdirectory (and depending on the events, perhaps on every file); doing it on a handful of directories with a few tens or hundreds of files should be no problem on a modern machine with lots of memory, but YMMV, so "Caveat Emptor"!

Now to have emacs do all this: run the above command as an asynchronous subprocess, watch its output and whenever it sees something interesting in that output, call the above function to re-evaluate org-agenda-files.

To start the process, you need to add something like the following bit of code to your init file (or evaluate it by hand if you want to test things out: C-x C-e after the closing paren in an emacs-lisp buffer will do it):

(setq ndk/inotify-org-agenda-process
   (make-process 
      :name "ndk-inotify-org-agenda"
      :command `("inotifywait"
                 "-e" "create"
                 "-e" "delete"
                 "-m" "-r" ,(expand-file-name "~/org"))
      :buffer (get-buffer-create "*ndk/inotify-org-agenda-process-buffer*")
      :connection-type 'pipe
      :stderr (get-buffer-create "*ndk/inotify-org-agenda-process-stderr*")))

This uses the low-level make-process to create a process that runs the given command. It arranges so that the stdout of the command is inserted into the process buffer *ndk/inotify-org-agenda-process-buffer* and its stderr is inserted into the *ndk/inotify-org-agenda-process-stderr* buffer.

If you evaluate this code and visit those two buffers, you should be able to see output from the command whenever you create or delete a file in the ~/org directory or its subdirectories (it will also produce output when you create or delete a subdirectory, which we'll have to filter out, since the ultimate aim is to detect creation or deletion of Org mode files).

We now have to add a process filter to the above, so that we can examine the output and act accordingly. Read the "Processes" chapter in the Emacs Lisp Manual with C-h i g(elisp)Processes. In particular, the section Receiving Output from Processes describes process filters.

To add a process filter, we define a filter function and change the call to make-process above to specify the filter:

(setq ndk/inotify-org-agenda-process
   (make-process 
      :name "ndk-inotify-org-agenda"
      :command `("inotifywait"
                 "-e" "create"
                 "-e" "delete"
                 "-m" "-r" ,(expand-file-name "~/org"))
      :buffer (get-buffer-create "*ndk/inotify-org-agenda-process-buffer*")
      :connection-type 'pipe
      :filter #'ndk/inotify-org-agenda-process-filter
      :stderr (get-buffer-create "*ndk/inotify-org-agenda-process-stderr*")))

(defun ndk/inotify-org-agenda-process-filter (proc txt)
  (when (buffer-live-p (process-buffer proc))
    (display-buffer (process-buffer proc))
    (with-current-buffer (process-buffer proc)
      (let ((begin (marker-position (process-mark proc))))
        ;; Insert the text, advancing the process marker.
        (goto-char (process-mark proc))
        (insert-before-markers txt)
        (set-marker (process-mark proc) (point))
        (goto-char (process-mark proc))
        ;; the above is pretty generic: the reason we need a custom
        ;; filter function is so that we can test whether the process
        ;; output indicates that we should refresh
        ;; `org-agenda-files'. We remember where the previous output
        ;; had ended in `begin'; so the new output is the region
        ;; between `begin' and the current `point'. We pass that
        ;; information to the checker and if it returns non-nil, we
        ;; refresh `org-agenda-files'.
        (if (ndk/update-org-agenda-file-list? begin (point))
            (ndk/refresh-org-agenda-files))))))

If you don't specify a process filter, a default one is used that adds the output of the process to the process buffer. Here we define a custom filter that does the same thing but adds a few wrinkles. Filter functions take two arguments: the process and the string that the process has output since the last time the filter was called. We remember where the previous output ended, i.e. where our new output should go: that's the process mark. We make sure to go to that marker before inserting the new output, pushing any markers forward as we do so. We then set the process mark to the end of the new text and make sure that we go there (this should be a no-op, but a little paranoia never hurt). That's fairly generic: just about any filter would do a similar thing (but since this is the first filter I've ever written, if there are any problems, this is where I would look first: I wonder if this simple filter is too simple, although I have not seen any problems with it - yet). Now for the particular stuff: given the beginning and end of the insertion, we can always grab it and pass it to the checker to see if there are new files named 'foo.org' that were created; if so, we call ndk/refresh-org-agenda-files to go get the new list (the latter function was defined above).

Although the checker should probably check e.g. that what was created or deleted was actually an Org file (and not e.g. a directory or a non-Org file), that's a bit involved and would make this long answer even longer. So I just provide a dummy implementation here which always returns t (i.e. whatever inotify reports, we always refresh org-agenda-files), and leave the "real" implementation to the interested reader: it's an exercise in parsing strings but does not have much to do with processes. Here's the dummy implementation:

(defun ndk/update-org-agenda-file-list? (begin end)
  " Check whether the contents of the buffer (assumed to contain the
output of the inotify process) in the region between `begin' and
`end', indicate that we need to refresh `org-agenda-files'."

  ;; This is a simple example that always returns `t' (i.e.*anything*
  ;; added to or removed from the directory will trigger a refresh of
  ;; `org-agenda-files').
  ;;
  ;; But you could add some filtering, e.g. grab the contents of the
  ;; buffer using `buffer-substring' and parse it to figure out if the
  ;; file(s) that were added or removed should affect the agenda list
  ;; (e.g. adding or removing a subdirectory or a file that does not
  ;; end in `.org' should not trigger a a refresh). There may be some
  ;; addditional complications as well: it is not necessarily true
  ;; that the output of the command is spat out to the filter function
  ;; in one piece, so it may be necessary to deal with partial
  ;; lines. The simple implementation sidesteps these complications
  ;; but they must be kept in mind in general.
  t)

The comment describes some details of a possible implementation and some caveats that might arise (although I have not seen them in practice).

If you want to kill the process and the associated buffers, here's a (rough-and-ready) function to do it:


(defun ndk/inotify-org-agenda-process-cleanup ()
  (interactive)
  (delete-process ndk/inotify-org-agenda-process)
  (let ((kill-buffer-query-functions nil))
    (kill-buffer "*ndk/inotify-org-agenda-process-buffer*")
    (kill-buffer "*ndk/inotify-org-agenda-process-stderr*")))

Invoke with M-x ndk/inotify-org-agenda-process-cleanup. It kills the process and the two buffers, no questions asked, but it leaves a lot to be desired, in particular error handling.

4
  • this is amazing... I learned more from this one post than I did in my first 2 years using emacs... you are awesome. I ended up writing a function and binding it to key. But the inotify idea is awesome, and i'm playing around with that now. I always forget inotify exists :) Jul 31 at 16:33
  • I am glad! I'm still not finished though, but when I do you should probably come back and reread it :-)
    – NickD
    Jul 31 at 17:41
  • I will definitely come back to re-read it. Jul 31 at 23:18
  • Added a simple process filter (that does not do much filtering :-) ). There's actually one more method that I may add at some point, intermediate between the "refresh on demand" method and the "inotify" method: you can run an idle timer every few seconds that calls the refresh function. It is much simpler than the "inotify" method but it does a lot more unnecessary work (but it's not much work in any case, so it could be an acceptable solution).
    – NickD
    Aug 5 at 5:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.