org-mode batch export: Missing syntax highlighting

Question

I'm having issues achieving any syntax highlighting in the results of the following command:

emacs nix.org --batch -f org-html-export-to-html --kill

I know this question has been asked a few times already in various forms, but none of the advertised solutions I've found have worked for me (linked below).

My Environment

• spacemacs (via emacs 25.3) on the develop branch, current as of 2018 January 29
• htmlize-20171017.141 as a dependency of the org layer
• Attempting to highlight source blocks of haskell, lisp, yaml, and nix modes

What I'm Seeing

Naively performing the command above gives many lines of:

Cannot fontify src block (htmlize.el >= 1.34 required)

I tried copy-pasting the htmlize.el found in my .emacs.d/elpa/.. and refering to it manually:

emacs nix.org --batch -l htmlize.el -f org-html-export-to-html --kill

...and this makes the "Cannot find..." errors go away, but the output is still not highlighted properly. Actually, the lisp block does have one of its keywords bolded, but nothing else looks right. Similar hacks to manually include lisp code from say haskell-mode can achieve bold/italics-only highlighting, which is not what I want.

--batch implies -q, which ignores all user config (to speed up start-up, I'm assuming). This seems to have the effect of ignoring all the Emacs packages I have installed (via spacemacs), and so the batch process can't see htmlize or any of my major modes to achieve proper colouring.

Questions

• Is there a way for me to tell a batch'd emacs process to respect my config and see all the packages I have? (this might tank each run of it though)
• Is there something else simple that I'm missing to achieve this highlighting properly?

My goal is to only commit my .org files for a blog, and have my server generate each .html properly on-the-fly during a deploy. I could just generate the nice .html files on my own machine and commit those, but I really don't want to do that.

Thanks!

Related

• https://stackoverflow.com/questions/10642888/syntax-highlighting-within-begin-src-block-in-emacs-orgmode-not-working
• https://stackoverflow.com/questions/24082430/org-mode-no-syntax-highlighting-in-exported-html-page
• https://stackoverflow.com/questions/31106417/no-syntax-highlighting-with-org-html-export-to-html-when-executing-with-systemd

Answer 1

1. You have already mentioned that emacs skips the personal initialization files if it is called with --batch. Therefore, you need to call package-initialize yourself if you want to use package.el.

2. htmlize-buffer uses the text properties added by font-lock to compile style information for the html buffer/file. Therefore, we need font-lock even if the files are processed in batch-mode.

3. The text properties carry the faces and the faces contain the color information for the text and the background.

4. htmlize-buffer calls face-attribute for determining the text color. That function calls in turn internal-get-lisp-face-attribute. That internal function always checks whether the frame admits colorizing. But, emacs does not have any frame in batch-mode. So we must override face-attribute to htmlize the color information we want.

The following lisp code can be used to run package-initialize and to override face-attribute. (At the end of this code there is also a little debugging tool print-args-and-ret that easies debugging emacs in batch-mode.)

(package-initialize)

(require 'font-lock)

(require 'subr-x) ;; for `when-let'

(unless (boundp 'maximal-integer)
  (defconst maximal-integer (lsh -1 -1)
    "Maximal integer value representable natively in emacs lisp."))

(defun face-spec-default (spec)
  "Get list containing at most the default entry of face SPEC.
Return nil if SPEC has no default entry."
  (let* ((first (car-safe spec))
     (display (car-safe first)))
    (when (eq display 'default)
      (list (car-safe spec)))))

(defun face-spec-min-color (display-atts)
  "Get min-color entry of DISPLAY-ATTS pair from face spec."
  (let* ((display (car-safe display-atts)))
    (or (car-safe (cdr (assoc 'min-colors display)))
    maximal-integer)))

(defun face-spec-highest-color (spec)
  "Search face SPEC for highest color.
That means the DISPLAY entry of SPEC
with class 'color and highest min-color value."
  (let ((color-list (cl-remove-if-not
             (lambda (display-atts)
               (when-let ((display (car-safe display-atts))
                  (class (and (listp display)
                          (assoc 'class display)))
                  (background (assoc 'background display)))
             (and (member 'light (cdr background))
                  (member 'color (cdr class)))))
             spec)))
    (cl-reduce (lambda (display-atts1 display-atts2)
         (if (> (face-spec-min-color display-atts1)
            (face-spec-min-color display-atts2))
             display-atts1
           display-atts2))
           (cdr color-list)
           :initial-value (car color-list))))

(defun face-spec-t (spec)
  "Search face SPEC for fall back."
  (cl-find-if (lambda (display-atts)
        (eq (car-safe display-atts) t))
          spec))

(defun my-face-attribute (face attribute &optional frame inherit)
  "Get FACE ATTRIBUTE from `face-user-default-spec' and not from `face-attribute'."
  (let* ((face-spec (face-user-default-spec face))
     (display-attr (or (face-spec-highest-color face-spec)
               (face-spec-t face-spec)))
     (attr (cdr display-attr))
     (val (or (plist-get attr attribute) (car-safe (cdr (assoc attribute attr))))))
    ;; (message "attribute: %S" attribute) ;; for debugging
    (when (and (null (eq attribute :inherit))
           (null val))
      (let ((inherited-face (my-face-attribute face :inherit)))
    (when (and inherited-face
           (null (eq inherited-face 'unspecified)))
      (setq val (my-face-attribute inherited-face attribute)))))
    ;; (message "face: %S attribute: %S display-attr: %S, val: %S" face attribute display-attr val) ;; for debugging
    (or val 'unspecified)))

(advice-add 'face-attribute :override #'my-face-attribute)

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;; Debugging:
(defmacro print-args-and-ret (fun)
  "Prepare FUN for printing args and return value."
  `(advice-add (quote ,fun) :around
           (lambda (oldfun &rest args)
         (let ((ret (apply oldfun args)))
           (message ,(concat "Calling " (symbol-name fun) " with args %S returns %S.") args ret)
           ret))
           '((name "print-args-and-ret"))))

; (print-args-and-ret htmlize-faces-in-buffer)
; (print-args-and-ret htmlize-get-override-fstruct)
; (print-args-and-ret htmlize-face-to-fstruct)
; (print-args-and-ret htmlize-attrlist-to-fstruct)
; (print-args-and-ret face-foreground)
; (print-args-and-ret face-background)
; (print-args-and-ret face-attribute)

Let us assume that the above elisp code is stored in ~/tmp/loadme.el and the org file which we want to htmlize is ~/tmp/test.org. In that case you would call emacs as follows to produce a html file ~/tmp/test.html:

emacs --batch --load ~/tmp/loadme.el ~/tmp/test.org --eval '(progn (outline-show-all) (font-lock-flush) (font-lock-fontify-buffer) (with-current-buffer (htmlize-buffer) (write-region (point-min) (point-max) "~/tmp/test.html")))'

Answer 2

I think the intended way to get syntax highlighting in Org Mode batch exports using htmlize is to:

1. First generate a CSS file with M-x org-html-htmlize-generate-css.
2. Then (setq org-html-htmlize-output-type 'css) during the batch export, and link the previously generated CSS file via org-html-head.

From the documentation of org-html-htmlize-output-type:

Choices are css to export the CSS selectors only,inline-css
to export the CSS attribute values inline in the HTML or nil to
export plain text.  We use as default inline-css, in order to
make the resulting HTML self-containing.

However, this will fail when using Emacs in batch mode for export, because
then no rich font definitions are in place.  It will also not be good if
people with different Emacs setup contribute HTML files to a website,
because the fonts will represent the individual setups.  In these cases,
it is much better to let Org/Htmlize assign classes only, and to use
a style file to define the look of these classes.
To get a start for your css file, start Emacs session and make sure that
all the faces you are interested in are defined, for example by loading files
in all modes you want.  Then, use the command
M-x org-html-htmlize-generate-css to extract class definitions.

Note: at some point in the future, syntax highlighting might be made to work the same way in batch exports as in regular sessions using the engrave-faces package, according to this post on the Org Mode mailing list.

Answer 3

This is an update to Tobias's answer to handle inherited faces:

(package-initialize)

(require 'font-lock)
(require 'cl-seq)
(require 'subr-x) ;; for `when-let'

(unless (boundp 'maximal-integer)
  (defconst maximal-integer (lsh -1 -1)
    "Maximal integer value representable natively in emacs lisp."))

(defun face-spec-default (spec)
  "Get list containing at most the default entry of face SPEC.
Return nil if SPEC has no default entry."
  (let* ((first (car-safe spec))
         (display (car-safe first)))
    (when (eq display 'default)
      (list (car-safe spec)))))

(defun face-spec-min-color (display-atts)
  "Get min-color entry of DISPLAY-ATTS pair from face spec."
  (let* ((display (car-safe display-atts)))
    (or (car-safe (cdr (assoc 'min-colors display)))
        maximal-integer)))

(defun face-spec-highest-color (spec)
  "Search face SPEC for highest color.
That means the DISPLAY entry of SPEC
with class 'color and highest min-color value."
  (let ((color-list (cl-remove-if-not
                     (lambda (display-atts)
                       (when-let ((display (car-safe display-atts))
                                  (class (and (listp display)
                                              (assoc 'class display)))
                                  (background (assoc 'background display)))
                         (and (member 'light (cdr background))
                              (member 'color (cdr class)))))
                     spec)))
    (cl-reduce (lambda (display-atts1 display-atts2)
                 (if (> (face-spec-min-color display-atts1)
                        (face-spec-min-color display-atts2))
                     display-atts1
                   display-atts2))
               (cdr color-list)
               :initial-value (car color-list))))

(defun face-spec-t (spec)
  "Search face SPEC for fall back."
  (cl-find-if (lambda (display-atts)
                (eq (car-safe display-atts) t))
              spec))

(defun my-face-attribute (face attribute &optional frame inherit)
  "Get FACE ATTRIBUTE from `face-user-default-spec'.
And not from `face-attribute'.
FRAME and INHERIT are ignored."
  (ignore frame)
  (ignore inherit)
  (let* ((face-spec (face-user-default-spec face))
         (display-attr (or (face-spec-highest-color face-spec)
                           (face-spec-t face-spec)))
         (attr (cdr display-attr))
         (val (or (plist-get attr attribute) (car-safe (cdr (assoc attribute attr))))))
    ;; Debugging
    ;; (message "attribute: %S" attribute)
    (when (and (null (eq attribute :inherit))
               (null val))
      (let ((inherited-face (my-face-attribute face :inherit)))
        (when (and inherited-face
                   (null (eq inherited-face 'unspecified)))
          (setq val (my-face-attribute inherited-face attribute))))
      (when (null val)
        ;; See if we have a default entry, (default :inherit FACE)
        ;; See font-lock-comment-delimiter-face
        (while face-spec
          (let ((entry (car face-spec)))
            (if (and (= (length entry) 3)
                     (eq (car entry) 'default)
                     (eq (car (cdr entry)) :inherit))
                (progn
                  (setq val (my-face-attribute (car (cdr (cdr entry))) attribute))
                  (setq face-spec nil))
              (setq face-spec (cdr face-spec)))))))

    ;; Debugging:
    ;; (message "face: %S attribute: %S display-attr: %S, val: %S" face attribute display-attr val)
    (or val 'unspecified)))

(advice-add 'face-attribute :override #'my-face-attribute)

;;; Faces Debugging

;; (defmacro print-args-and-ret (fun)
;;   "Prepare FUN for printing args and return value."
;;   `(advice-add (quote ,fun) :around
;;            (lambda (oldfun &rest args)
;;          (let ((ret (apply oldfun args)))
;;            (message ,(concat "Calling " (symbol-name fun) " with args %S returns %S.") args ret)
;;            ret))
;;            '((name "print-args-and-ret"))))
;;
;; (print-args-and-ret htmlize-faces-in-buffer)
;; (print-args-and-ret htmlize-get-override-fstruct)
;; (print-args-and-ret htmlize-face-to-fstruct)
;; (print-args-and-ret htmlize-attrlist-to-fstruct)
;; (print-args-and-ret face-foreground)
;; (print-args-and-ret face-background)
;; (print-args-and-ret face-attribute)