Tangle Org file containing #+INCLUDE directives and multiple tangle targets

Question

Context

I'm using org-babel-tangle to generate my configuration from an Org file config.org. Recently, I did some clean-up by

1. dispatching config.org to multiple Org files config/*.org;
2. replace the content of config.org with multiple #+INCLUDE: config/*.org directives.

To illustrate a bit, the directory tree is as follows:

config/
|______ A.org
|______ B.org
|______ C.org
|
config.org

where

## config.org
#+INCLUDE: config/A.org
#+INCLUDE: config/B.org
#+INCLUDE: config/C.org
...

and

## [ABC].org
#+BEGIN_SRC emacs-lisp :tangle target1.el
;; elisp code
#+END_SRC

#+BEGIN_SRC emacs-lisp :tangle target2.el
;; elisp code
#+END_SRC
...

Issue & attempt

However, I ran into the issue that org-babel-tangle doesn't handle #+INCLUDE directives. So, I follow the idea of this post which consists of

1. exporting config.org to an Org file config-export.org with org-export-to-file;
2. org-babel-tangle the exported Org file config-export.org to tangle targets.

This approach does replace the multiple #+INCLUDE directives of config.org by the content of config/*.org, but all tangle options are erased. Namely, the exported config-export.org looks like this (all emacs-lisp tangle target filename are lost):

## config-export.org
#+BEGIN_SRC emacs-lisp
;; elisp code for target1.el
#+END_SRC

#+BEGIN_SRC emacs-lisp
;; elisp code for target2.el
#+END_SRC
...

Question

How to org-babel-tangle an Org file with #+INCLUDE directives as it is intended?

(i.e. (1) include all Org files (preserving :tangle target.el), then (2) org-babel-tangle the whole Org file.)

Related question

A similar question has been raised in the Org mode mailing-list.

Answer 1

I don't know how to do this without changing the code (it might be possible with org-babel-pre-tangle-hook but it did not seem at all simple, so I didn't try that). But as soon as you accept code changes, it does not seem too bad.

The simplest possible way I could think of was to emulate in org-babel-tangle what org-export-as is doing WRT included files. A simple test (similar to what you describe above) succeeded easily (but see caveats at the end), so if you want to try replicating the change, I include the patch below.

The patch itself would look big, but that's only because I'm taking all of the code of org-babel-tangle and enclosing it in another layer, which changes the indentation and that's most of the change. Ignoring whitespace changes, the "real" change is much smaller: I got the patch below with git diff -b in order to ignore whitespace. You can then see that it is very small: we just enclose almost the whole body of org-babel-tangle in (org-export-with-buffer-copy ...), so that we can do transformations to the text in a temp buffer and not disturb the original and then add one more transformation to process the #+INCLUDE: directives, before the resulting buffer is then tangled:

diff --git a/lisp/ob-tangle.el b/lisp/ob-tangle.el
index b74b3fa0c..5e6eebf63 100644
--- a/lisp/ob-tangle.el
+++ b/lisp/ob-tangle.el
@@ -206,6 +206,9 @@ export file for all source blocks.  Optional argument LANG-RE can
 be used to limit the exported source code blocks by languages
 matching a regular expression."
   (interactive "P")
+  (org-export-with-buffer-copy
+   (org-export-expand-include-keyword)
+
    (run-hooks 'org-babel-pre-tangle-hook)
    ;; Possibly Restrict the buffer to the current code block
    (save-restriction
@@ -306,7 +309,7 @@ matching a regular expression."
     (mapc (lambda (pair)
         (when (cdr pair) (set-file-modes (car pair) (cdr pair))))
           path-collector)
-   (mapcar #'car path-collector)))))
+    (mapcar #'car path-collector))))))
 
 (defun org-babel-tangle-clean ()
   "Remove comments inserted by `org-babel-tangle'.

That produces the expected results WRT the tangled files I think, but you should check to make sure that it does the right thing in your case: my tests are toy tests so they might miss tricky situations that might arise in your "real world" tests.

There is one problem which I don't understand yet: the config.org buffer itself is changed in my tests, despite the org-export-with-copy-buffer, so at the end of the tangle I have to undo the change with C-x u and save it. That's probably because I don't understand what org-export-with-copy-buffer is really doing, but I have not checked that code yet.

OTOH, if that is resolved and the tests all pass, we can propose this as an enhancement to the Org mode mailing list and put that problem (from 2010!) behind us.

EDIT: I don't see any problem with org-babel-tangle-file - it calls org-babel-tangle underneath, so in theory it should do the (almost) right thing and testing it with a simple example works fine for me. Maybe you can post a (small) example that gives you problems by editing your question?

I'm also testing with the following workaround to get around the issue of config.org getting changed:

#+begin_src emacs-lisp
(copy-file "./config.org.orig" "./config.org")
(org-babel-tangle-file "./config.org")
(delete-file "./config.org")
#+end_src

Ugly, but it seems to work (except that I have to answer the "changed on disk" question).

Answer 2

Method 1

Manual method using default org-mode keybindings

Setup

1. Add Org to Org Babel Load Languages configuration and then apply and save new configuration.

2. Add #+EXPORT_FILE_NAME: keyword near top of config.org.

   e.g.

   #+EXPORT_FILE_NAME: temp.org
   

3. Add #+PROPERTY: header-args:org :results drawer replace keyword near top of config.org.

   e.g.

   #+PROPERTY: header-args:org :results drawer replace
   

4. Update #+INCLUDE: to include the src org syntax.

   e.g.

   #+INCLUDE: config/A.org src org
   #+INCLUDE: config/B.org src org
   #+INCLUDE: config/C.org src org
   

5. Add :eval never header to each of SRC blocks that will be tangled inside the included config/*.org files.

   e.g.

   #+BEGIN_SRC elisp :tangle tangle1.el :eval never 
     ;; Code goes here
   #+END_SRC
   

Manual Execution Steps

1. Open config.org in emacs.

   The config.org file should look similar to the example below:

   #+EXPORT_FILE_NAME: temp.org
   #+PROPERTY: header-args:org :results drawer replace 
   
   #+INCLUDE: config/A.org src org
   #+INCLUDE: config/B.org src org
   #+INCLUDE: config/C.org src org
   

2. Export config.org using C-c C-e O v key chord. This will create a new temp.org file and open in emacs.

   The temp.org file should look similar to the example below:

   #+export_file_name: temp.org
   #+property: header-args:org :results drawer replace
   
   #+begin_src org
     ,#+BEGIN_SRC elisp :tangle tangle1.el :eval never 
       ;; Code goes here
     ,#+END_SRC
   #+end_src
   #+begin_src org
     ,#+BEGIN_SRC elisp :tangle tangle2.el :eval never 
       ;; Code goes here
     ,#+END_SRC
   #+end_src
   #+begin_src org
     ,#+BEGIN_SRC elisp :tangle tangle3.el :eval never 
       ;; Code goes here
     ,#+END_SRC
   #+end_src
   

3. Execute all the blocks in the temp.org using C-c C-v C-b key chord.

   The temp.org file should look similar to the example below:

   #+export_file_name: temp.org
   #+property: header-args:org :results drawer replace
   
   #+begin_src org
     ,#+BEGIN_SRC elisp :tangle tangle1.el :eval never 
       ;; Code goes here
     ,#+END_SRC
   #+end_src
   
   #+RESULTS:
   :results:
   #+BEGIN_SRC elisp :tangle tangle1.el :eval never 
     ;; Code goes here
   #+END_SRC
   :end:
   
   #+begin_src org
     ,#+BEGIN_SRC elisp :tangle tangle2.el :eval never 
       ;; Code goes here
     ,#+END_SRC
   #+end_src
   
   #+RESULTS:
   :results:
   #+BEGIN_SRC elisp :tangle tangle2.el :eval never 
     ;; Code goes here
   #+END_SRC
   :end:
   
   #+begin_src org
     ,#+BEGIN_SRC elisp :tangle tangle3.el :eval never 
       ;; Code goes here
     ,#+END_SRC
   #+end_src
   
   #+RESULTS:
   :results:
   #+BEGIN_SRC elisp :tangle tangle3.el :eval never 
     ;; Code goes here
   #+END_SRC
   :end:
   

4. Execute the tangle in the temp.org using C-c C-v C-t key chord. This will tangle the imported SRC blocks from the config/*.org files.

   This will create the tangled files similar to the files listing below:

   config.org
   tangle1.el
   tangle2.el
   tangle3.el
   temp.org
   
   ./config:
   A.org
   B.org
   C.org
   

---

This answer was tested using:
emacs version: GNU Emacs 27.1
org-mode version: 9.3.7

Answer 3

I am not sure if it still helps and if it is exactly what you need... I am using something like within spacemacs:

  ;;-----------------------------------------------------------------------
  (message "--- START : filemanage ----")
  (let ((srcuf (concat dotspacemacs-directory "user_filemanage.org"))
        (uf (concat dotspacemacs-directory "user-filemanage.el"))
        )
    (when (file-newer-than-file-p srcuf uf)

      (message "--- File is adjusted : filemanage ----")
      (call-process
       (concat invocation-directory invocation-name)
       nil nil t
       "-q" "--batch" "--eval" "(require 'ob-tangle)"
       "--eval" (format "(org-babel-tangle-file \"%s\")" srcuf)))
    )

  (message "--- DONE : filemanage ----")
  ;;-----------------------------------------------------------------------
  (message "--- START : helm ----")
  (let ((srcuh (concat dotspacemacs-directory "user_helm.org"))
        (uh (concat dotspacemacs-directory "user-helm.el"))
        )
    (when (file-newer-than-file-p srcuh uh)

      (message "--- File is adjusted : helm ----")
      (call-process
       (concat invocation-directory invocation-name)
       nil nil t
       "-q" "--batch" "--eval" "(require 'ob-tangle)"
       "--eval" (format "(org-babel-tangle-file \"%s\")" srcuh)))
    )

And for loading within the config section

  (let ((uf (concat dotspacemacs-directory "user-filemanage.el")))
    (load-file uf))
  (let ((uh (concat dotspacemacs-directory "user-helm.el")))
    (load-file uh))

and in user_filemanage.el I have something like

*** dired-open
#+BEGIN_SRC emacs-lisp :tangle user-filemanage.el
;; dired open.............................................................................
(require 'dired-open)  
#+END_SRC