Wildcard in org-agenda-files

Question

I have a project base folder, ~/prj, inside of which I have many individual projects, 2014_prj1, 2014_prj2, ..., inside of which I have, among others, a doc folder. The layout looks like this:

~/prj
~/prj/2014_prj1
~/prj/2014_prj1/README.org
~/prj/2014_prj1/data
~/prj/2014_prj1/doc
~/prj/2014_prj2
~/prj/2014_prj2/README.org
~/prj/2014_prj2/data
~/prj/2014_prj2/doc

Now I'd like to add all project directories (2014_prjX) to org-agenda-files, as well as all doc subdirectories of the project directories. The data folders should not be part of org-agenda-files, because they can possibly contain 10000s of files.

How can I achieve this? I tried a simple

(setq org-agenda-files (quote ("~/doc/notes"
                               "~/.emacs.d"
                               "~/prj/*/doc"
                               )
                        )

but this gives me the error

Non-existing agenda file ~/prj/*/doc

Answer 1

You can use the function file-expand-wildcards to get all the files matching the wildcard add them to org-agenda-files variable. Something like the following

(setq org-agenda-files (append '("~/doc/notes" "~/.emacs.d") (file-expand-wildcards "~/prj/*/doc")))

Answer 2

I created a more generic approach, which allows you to have deeper nesting. For example, I have a rich tree of notes and some folders have tasks.org file in it. This code will collect all of the fly:

(defun set-org-agenda-files ()
  (require 'f)
  (require 's)

  (setq org-agenda-files
    (f-entries org-directory
               (lambda (filename)
                 (s-ends-with-p "/tasks.org" filename))
               t)))