How to achieve dynamic projects without fixed paths for publishing from Org mode?

Question

One usually adds entries to org-publish-project-alist to describe what and how needs to be published, including :publishing-directory, fancy :publishing-function and alike. However it seems handy for some projects to reside in version controlled repository with all that it entails, i.e. different users, different checkout paths. My problem is that I cannot wrap my head around how would I go about making it all flexible.

I'd like everything to go by a "standard" project name no matter what repository clone I'm in, i.e. if I have multiple cloned copies of my repository, I'd like to have same project name while exporting files from subfolders. And I'd like for an export to take place, e.g. to a sibling folder, i.e. not some hard-coded paths.

It seems that .dir-locals.el is the way to go. But it is getting gnarly with on the fly evaluation of various path names, etc and I have to mark variable and values as safe. But minor changes cause pollution of safe-local-variable-values in my custom-file.

Is there a canonical way on how to achieve such setups in general?

Update: Currently, I'm having a custom function that overrides quite many things (starting with looking up a working repository copy root containing .git) on a global scale with setq and this function is wrapped into a minor mode. It is still somewhat broken (autoload part) due to my knowledge (or lack thereof) of Elisp so I have to re-evaluate the file as some variables aren't set right. And a huge shortcoming is that I have to restart Emacs to get variables back to their original values as it is too many of them that I change (like org-twbs-tag-class-prefix, org-twbs-format-headline-function, org-latex-classes, org-babel-default-header-args:dot and more).

Answer 1

This is going to be a long post, so bear with me. The only Emacs way I see so far is to use .dir-locals.el and keep it relatively clean so you don't have to approve evals every time you change something. Presumably you trust your repos:-) The rest is to give you an idea... from newb to newbs.

((org-mode
  (eval . (my-mega-project-setup))
  ))

You got to define that function in dot emacs for "default" case. In most cases lisp code becomes stable fast, and only whatever you do changes from location to location on your filesystem. I mean for me it was not a big deal to hard code lisp path for interactive use.

(add-to-list 'load-path "~/Dropbox/my-mega-project/lisp")
(autoload 'my-mega-project-setup "megaproject")

Of course you got to customize safe-local-variable-values with (eval my-mega-project-setup).

I have approximately the following hierarchy in my repo:

root + topic1/
     + topic2/
     | ...
     + common - org_setup_template.org
     + assets - less, js stuff
     + lisp + batch.el
     |      + megaproject.el
     + publish.cmd
     + package.json with webpack and such for nodejs package manager

Let's move now to "dynamic" part. The key here is to define lots of stuff you want to be buffer-local in your ./lisp/megaproject.el. Mine is really huge at this point so I'll include only an excerpt to give you an idea. It is not meant to be final, e.g. missing functions etc.

(require 'org)
(require 'ox-html)
(require 'ox-latex)
(require 'ox-md)
(require 'ox-publish)
(require 'ox-twbs)
(require 'ox-odt)
(require 'ob-dot)
(require 'cl-lib)     ; cl-intersection to test for excluded headlines

(defconst odt-schema-dir
  "c:/ProgramData/Chocolatey/lib/emacs64/tools/emacs/share/emacs/24.5/etc/schema"
  "You might need to copy od-manifest-schema-v1.2-os.rnc & od-schema-v1.2-os.rnc off the web into there.")

(defconst my-directory
  (let*                               ; TODO: search for .git/ instead
      ((d (dir-locals-find-file ".")))
    (file-name-directory
     (if d
         (if (stringp d) d (car d))
       (directory-file-name
        (file-name-directory load-file-name)))))
  "Absolute path to root folder.")
...
(defun my-mega-project-setup ()
  "Called from .dir-locals.el to initialize things for the project."
  (set (make-local-variable 'org-twbs-format-headline-function) #'my-format-headline)
  (set (make-local-variable 'org-confirm-babel-evaluate) nil)
  (advice-add 'org-export--prepare-file-contents :around #'my-export--prepare-file-contents)
  (set (make-local-variable 'org-latex-classes)
       (cons `("my"
               ,my-latex-head
               ("\\section{%s}" . "\\section*{%s}")
               ("\\subsection{%s}" . "\\subsection*{%s}")
               ("\\subsubsection{%s\}" . "\\subsubsection*{%s}")
               ("\\paragraph{%s}" . "\\paragraph*{%s}")
               ("\\subparagraph{%s}" . "\\subparagraph*{%s}")) org-latex-classes))
  (set (make-local-variable 'org-babel-default-header-args:dot)
       (append '((:exports . "results")
                 (:cache . "yes")
                 (:cmdline . "-Tpng -Gbgcolor=transparent"))
               org-babel-default-header-args:dot))
  (set (make-local-variable 'org-babel-default-header-args:sh)
       (append '((:exports . "both")
                 (:cache . "yes")
                 (:results . "verbatim"))
               org-babel-default-header-args:sh))
  (set (make-local-variable 'org-use-property-inheritance) t)
  (set (make-local-variable 'org-latex-listings) 'minted)
  (set (make-local-variable 'org-latex-pdf-process) '("latexmk -shell-escape -pdf %f"))
  (set (make-local-variable 'org-publish-timestamp-directory)
       (concat my-directory ".org-timestamps/"))
  (set (make-local-variable 'my-publishing-directory) ; otherwise needs to advise org-publish
       (expand-file-name "my-out"
                         (file-name-directory
                          (directory-file-name my-directory))))
  (set (make-local-variable 'org-structure-template-alist) (cons '("a" "#+BEGIN_assert\n?\n#+END_assert") org-structure-template-alist))
  (set (make-local-variable 'org-publish-project-alist) ;;; <<--- MAGIC HAPPENS HERE in alike statements
       `(("my-notes"
          ;; :makeindex t
          :base-directory ,my-directory
          :base-extension "org"
          :publishing-directory my-publishing-directory
          :recursive t
          :exclude "common\\|lisp\\|\\(setup\\|snippets\\|all\\|readme\\)\\.org"
          :publishing-function ((lambda (plist filename pub-dir)
                                  (let ((org-twbs-tag-class-prefix "label label-default ")
                                        (org-twbs-htmlize-output-type 'css)
                                        (org-twbs-footnotes-section my-footnotes-section)
                                        (org-twbs-todo-kwd-class-prefix "label label-default "))
                                    (org-publish-org-to 'my filename ".html" plist pub-dir))))
          :headline-levels 4
          :table-of-contents t
          :drawers nil
          :html-doctype "html5"
          :html-html5-fancy t
          :html-metadata-timestamp-format "%Y-%m-%d %a"
          :html-head-include-default-style nil
          :html-head-include-scripts nil
          :creator-info nil
          :html-preamble my-breadcrumbs
          :html-postamble ,my-postamble
          ;; :html-lightbox-enabled t        ; experimental patch https://github.com/marsmining/ox-twbs/issues/19
          :auto-sitemap nil ; t
          )
....

So far we have a usable interactive setup. Now we need to furnish command line publishing stuff that would work from any location you checked your repo to.

I have a batch file for the outer layer

@echo off
rem Due to variation in user setups, consider this a sample file.
rem Modify to your needs and use it leaving this file unmodified.

rem Path to pygments for syntax highlighting in PDFs
path C:\Miniconda3\envs\python27\Scripts;%PATH%
rem R for reproducible examples
path %ProgramFiles%\R\R-3.2.3\bin\i386;%PATH%
rem Where to find emacs
path %ChocolateyInstall%\lib\emacs64\tools\emacs\bin;%PATH%
rem LibreOffice's soffice path for odt to docx conversion
path %ProgramFiles%\LibreOffice 5\program;%PATH%

emacs -Q --batch --script lisp/batch.el > publish.log 2>&1

And, finally, ./lisp/batch.el

;; This is an alternative emacs init script to be used for batch publishing,
;; i.e. emacs -Q --batch --script <this-file.el>

(defun ensure-org-above-8 ()
  "Emacs comes with Org 7.9.3.
We can get Org 8+ from ELPA if user installed it."
  (require 'package)
  (add-to-list 'package-directory-list "~/Dropbox/Personal/emacs/.emacs.d/elpa")
  (setq package-enable-at-startup nil)
  (package-initialize)
)
(ensure-org-above-8)
(require 'org)
(message "Using Org version: %s" org-version)
(if (string< org-version "8.3") (error "Incorrect version of Org is loaded!"))

(require 'ess-site)                     ; R code fontification

(require 'ob-plantuml)
(setq org-plantuml-jar-path (concat (file-name-directory load-file-name) "plantuml.jar"))
(require 'ob-ditaa)
(setq org-ditaa-jar-path (concat (file-name-directory load-file-name) "ditaa0_9.jar"))

(add-to-list 'load-path (file-name-directory load-file-name))
(require 'megaproject)
(load-theme 'zenburn t)
(add-to-list 'safe-local-eval-forms '(my-mega-project-setup))
(find-file "index.org")
;; (update-all-in-one-org)
(org-publish-project "all")

Answer 2

According to the description of (org-publish) function:

(org-publish PROJECT &optional FORCE ASYNC)

Publish PROJECT.

PROJECT is either a project name, as a string, or a project alist (see `org-publish-project-alist' variable).

So I created an emacs-lisp-script following this scheme:

#!/usr/bin/emacs --script

(package-initialize)
(require 'org)

(setq subprojects '(("foobar"
             :base-directory "./files/"
             :base-extension "org"
             :publishing-directory "./_pages/"
             :publishing-function org-html-publish-to-html
             ;; :exclude "regex-stuff to exlude"
             :headline-levels 3
             :section-numbers nil
             :with-toc t)

            ("images"
             :base-directory "./images/"
             :base-extension "jpg\\|gif\\|png"
             :publishing-directory "./_pages/images/"
             :publishing-function org-publish-attachement)))

(mapc (lambda (project) (org-publish project t)) bar)

This works quite well for me so far, except that you need to publish from command-line and can't do from the buffer anymore.

Only thing I'd like to improve is to be able to pass in the FORCE parameter via command line and not having to edit the file all the time.