How can I walk an org-mode tree?

Question

Background

I'm writing a presentation mode for Emacs. I'd like the input to be org files, as org files are great for data.

Problem

I have to convert the org mode file into a list of "slide" data structures that I can iterate through. To do this, I'd like to take something like the following org-mode file:

* this is the first headline, with a title property and no contents
* this is the second headline, with contents
- dash list nested under the second headline
  - further nested
** nested headline

and be able to walk it. I've tried (org-element-parse-buffer), and that does give me a list of elements, but it's hard to figure out how to walk further in them. For example, calling (org-element-map (org-element-parse-buffer) 'headline #'identity) gives a list of three elements; the last one represents the "nested headline". I want "nested headline" to be a child of "this is the second headline, with contents".

Avoiding the XY problem

I'm certainly open to other ways of converting an org-mode file into an Elisp data structure. I don't think org-export is the right tool for me, because I don't want to end up with a new file containing the results, but a data structure I can iterate through. My naive way is something like "give me all the top-level headlines, and then I can get their properties and contained elements (e.g., plain text or nested lists -- whether further headlines or dash lists)".

Answer 1

I had a similar problem, so maybe this will help - I'm not very familiar with org export or org internals, but I couldn't find anything that would parse an org file to a tree structure. But given a buffer like

* england
** london
** bristol
* france

it will give you

(org-get-header-tree) => ("england" ("london" "bristol") "france")

and can include other information from the tree as well.

---

So given a flat list of levels we need to produce a tree, e.g. (1 1 2 3 1) => (1 1 (2 (3)) 1). I couldn't find a function that would do this either so wrote one after much drawing of cons cells - I'm sure there's a better way of doing this but it works. The function unflatten takes a flat list and couple of functions to extract the information you want from the list and the item levels and produces a tree structure.

In org-get-header-list you could add more information you want to extract from each item with calls to org-element-property, and then in org-get-header-tree you could include functions to extract the information from the list.

As it stands this doesn't include handling for dash lists, but maybe it could be adapted to handle those also without too much trouble...

(defun unflatten (xs &optional fn-value fn-level)
  "Unflatten a list XS into a tree, e.g. (1 2 3 1) => (1 (2 (3)) 1).
FN-VALUE specifies how to extract the values from each element, which
are included in the output tree, FN-LEVEL tells how to extract the
level of each element. By default these are the `identity' function so
it will work on a list of numbers."
  (let* ((level 1)
         (tree (cons nil nil))
         (start tree)
         (stack nil)
         (fn-value (or fn-value #'identity))
         (fn-level (or fn-level #'identity)))
    (dolist (x xs)
      (let ((x-value (funcall fn-value x))
            (x-level (funcall fn-level x)))
        (cond ((> x-level level)
               (setcdr tree (cons (cons x-value nil) nil))
               (setq tree (cdr tree))
               (push tree stack)
               (setq tree (car tree))
               (setq level x-level))
              ((= x-level level)
               (setcdr tree (cons x-value nil))
               (setq tree (cdr tree)))
              ((< x-level level)
               (while (< x-level level)
                 (setq tree (pop stack))
                 (setq level (- level 1)))
               (setcdr tree (cons x-value nil))
               (setq tree (cdr tree))
               (setq level x-level)))))
      (cdr start)))

; eg (unflatten '(1 2 3 2 3 4)) => '(1 (2 (3) 2 (3 (4))))


(defun org-get-header-list (&optional buffer) 
  "Get the headers of an org buffer as a flat list of headers and levels.
Buffer will default to the current buffer."
  (interactive)
  (with-current-buffer (or buffer (current-buffer))
    (let ((tree (org-element-parse-buffer 'headline)))
      (org-element-map 
          tree 
          'headline
        (lambda (el) (list 
                 (org-element-property :raw-value el) ; get header title without tags etc
                 (org-element-property :level el) ; get depth
                 ;; >> could add other properties here
                 ))))))

; eg (org-get-header-list) => (("pok" 1) ("lkm" 1) (("cedar" 2) ("yr" 2)) ("kjn" 1))


(defun org-get-header-tree (&optional buffer)
  "Get the headers of the given org buffer as a tree."
  (interactive)
  (let* ((headers (org-get-header-list buffer))
         (header-tree (unflatten headers  
                 (lambda (hl) (car hl))  ; extract information to include in tree
                 (lambda (hl) (cadr hl)))))  ; extract item level
    header-tree))

; eg (org-get-header-tree) => ("pok" "lkm" ("cedar" "yr") "kjn")

Answer 2

Parsing an Org-mode file requires knowing two things: the structure of the OrgNode, and how to recursively walk a tree. OrgNodes come in two forms:

1. (element-type (plist-of-properties) (child-1 OrgNode) (child-2 OrgNode) ... (child-n OrgNode))

or

2. #("secondary-string" beg end (plist) ...)

The org-element.el function org-element-parse-buffer returns an OrgNode root, org-data, which can be recursively walked. To do this, use the org-element.el function org-element-contents to return a list of child OrgNodes which can be iterated through very easily.

The difficulty in getting values out of an OrgNode is knowing that sometimes it is an OrgNode and sometimes a secondary string. If you don't test for the string, your code will often throw an error.

This code has three parts:

1. Obtain an OrgNode and parse some of its contents
2. Print some of the contents
3. Recurse though the OrgNode child nodes using the Preorder Traversal algorithm

You can of course obtain whatever values out of the OrgNode you specifically need and print them however you want. The Preorder Traversal algorithm is straight out of a textbook, but requires that you check for a list so as not to try to recurse into a secondary string, which will simply throw an error.

Once you see how an OrgTree is constructed, and the OrgNodes look, then the org-element.el mapping function org-element-map becomes easy to use.

(defun org-tree-traversal (org-node level)
   "Call this function with ORG-NODE being a completely parsed buffer
     from org-element-parse-buffer and LEVEL being 0 (zero)."

    ;; I. PARSE AN ORG-NODE CONTENTS

    (let* 
      ((type     (org-element-type org-node))
       (class    (org-element-class org-node))
       (contents (org-element-contents org-node))
       (props ; must be careful not to parse into a secondary string here
         (if (consp org-node)
            (second org-node) ; the properties exist and are a plist
            org-node)) ; the org-node is a secondary string
       (value (or (plist-get props :raw-value)
          (plist-get props :value)
          (and (stringp props) ; now get the string value for string
            (string-trim props))))
        (key (when
            (and (consp props) ; make sure props is a plist, not a string
                 (not (null value)))
             (let ((key (plist-get props :key)))
                (setf value (format "%s: %s" (if key key "VALUE") value)))))
        (level-indent (make-string level 32))
        (level-indent-dot (make-string level ?.)))

     ;; II. PRINT THE ORG-NODE CONTENTS

     (princ
          (format "%2d]%s%s (%s) %s\n"
           level
           level-indent-dot
           type
           class
           (unless
              (stringp props)
             (_prop-keys props)))) ; this is a utility function that prints
                                   ; all of the keys from the plist as a
                                   ; string; it can be omitted.
      (when value
         (princ (format "   %s%s\n" level-indent value)))
      (terpri)

 ;; III. RECURSE THROUGH THE ORG-NODE CHILD NODES

 (if (listp  contents) ; don't try to recurse into a secondary string
  (let ((child (first contents))
        (children (rest contents)))
    (while child
      (org-tree-traversal child (1+ level)) ; recurse
      (setf child (first children))
      (setf children (rest children))))))

t) ; RETURN TRUE WHEN DONE

(defun walk-org-tree (org-buf)
 "Walk an OrgTree from the Org buffer `ORG-BUF'.
Default is the current buffer."

 (interactive "bBuffer to parse: ")
 (with-temp-buffer-window "*OrgTree*" nil nil
  (org-tree-traversal
    (parse-org-buffer org-buf) 0)))

;; USAGE: (walk-org-tree "<org-buffer>.org")
;; or
;;        M-x walk-org-tree <RET> [<org-buffer>.org] | <RET> for current buffer

Answer 3

I made a very simple minor-mode for 'walking' through an org file. It doesn't require parsing the buffer at all, it just provides some shortcuts for moving from one narrowed subtree to the next/previous subtree.

(defun tws-present ()
  (interactive)
  (org-narrow-to-subtree))

(defun tws-present-next (ARG)
  (interactive "p")
  (beginning-of-buffer)
  (widen)
  (outline-forward-same-level ARG)
  (org-narrow-to-subtree))


(defun tws-present-previous (ARG)
  (interactive "p")
  (beginning-of-buffer)
  (widen)
  (outline-backward-same-level ARG)
  (org-narrow-to-subtree))

(defvar org-demo-minor-mode-map
  (let ((map (make-sparse-keymap)))
    (define-key map (kbd "C-<down>") 'tws-present-next)
    (define-key map (kbd "C-<up>") 'tws-present-previous)
    (define-key map (kbd "C-c C-s") 'tws-present)
    (define-key map (kbd "C-c C-q") 'org-demo-minor-mode)
    map))

(define-minor-mode org-demo-minor-mode
  "A minor mode for presenting org files as slides.
Very basic - narrows to current subtree, and provides navigation keys for
moving forwards and backwards through the file. Should probably also allow
for increasing font size?"
  nil nil
  :lighter " DEMO"
  (if org-demo-minor-mode
      (org-narrow-to-subtree)
    (widen))
  )