Speed up org-map-entries when matching by property

Question

Question: Why is org-map-entries property matching so slow, and what can I do to speed it up?

Background: I have a relatively simple use for org-map-entries: grab the effort (in integer minutes) from all org agenda entries with tag goal and a given priority (e.g. B).

(org-map-entries #'hw-org-get-effort-in-minutes "goal+PRIORITY=\"B\"" 'agenda)

This is terribly slow, taking over a minute for my ~12k line agenda file.

However, if I remove the PRIORITY from the filter so any goals tagged item is selected, it completes almost instantly.

I can also set filters like goal/DONE and they complete very quickly, but if I do something like goals+EFFORT>0 we are back to taking over a minute. It seems properties in general are very slow to match.

I found a cheat workaround: I can match properties inside the the mapped function very quickly using org-entry-get. When I do this, execution is less than a second. This seems silly, hopefully there is a better way, but at least it works!

Already tried: Since (benchmark 1000 (hw-org-effort-to-minutes "1:20")) returns "Elapsed time: 0.000019s", I don't think my function contributes much.

According to profiler, ~40% of CPU time is used by cond, with ~29% coming from element parsing (org-element--current-element). The next two largest contributions overall are 14% and 13%, so the 40% of cond seems to be the bulk of the problem. Not sure why element parsing would be done more often with property matchers, unless the difference comes from parsing only header (tags, TODO) vs. header + body (properties).

Answer 1

One way to improve speed is to parse the contents of your agenda files once in a temporary buffer collecting the effort of all the entries matching goal+PRIORITY="B" (see Test 1). With ~10K lines, I get "Elapsed time: 0.052280s" compared to "Elapsed time: 1.340006s" using org-map-entries (Test 2) which I think is what you were trying to do. For better results using org-map-entries you can try Test 3, which is also pretty fast. Tested with Emacs version 26.2 and Org mode version 9.2.4.

Test 1 (fastest)

(org-duration-from-minutes
 (apply '+ (let (efforts
                 (regexp (concat org-effort-property ":\s*\\(.+\\)")))
             (with-temp-buffer
               (mapcar #'insert-file-contents org-agenda-files)
               (goto-char (point-min))
               (while (re-search-forward regexp nil t)
                 (let ((effort (match-string 1)))
                   (save-excursion
                     (outline-previous-heading)
                     (when (and (member "goal" (org-get-tags))
                                (= (and (looking-at org-heading-regexp)
                                        (org-get-priority (match-string 0)))
                                1000))
                    (push (org-duration-to-minutes effort) efforts))))))
          efforts)))

Test 2 (slowest)

(org-duration-from-minutes
 (apply '+ (org-map-entries
            (lambda ()
              (org-duration-to-minutes
               (org-entry-get nil org-effort-property)))
            "goal+PRIORITY=\"B\""
            'agenda)))

Test 3 (pretty good)

(org-duration-from-minutes
 (apply '+ (org-map-entries
            (lambda ()
              (if (re-search-forward (concat org-effort-property ":\s*\\(.+\\)")
                                     (save-excursion
                                       (org-end-of-meta-data)
                                       (point))
                                     t)
                  (let ((effort (match-string 1)))
                    (outline-previous-heading)
                    (when (looking-at org-complex-heading-regexp)
                      (let ((priority (match-string 3))
                            (tags (match-string 5)))
                        (if (and (string= priority "[#B]")
                                 (string-match ":goal:" tags))
                            (org-duration-to-minutes effort)
                          0))))
                0))
            nil 'agenda)))