Org-babel: Notify if the result of execution differs from the previous recorded result

Question

I want to use my org-mode documentation as a sort of quasi-unittest; I want to execute the whole buffer, and be notified if marked cells' new results differ from their previous results. E.g,

#+begin_src python :results verbatim :exports both :wrap example :notify_on_change
x3(10)
#+end_src

#+RESULTS:
#+begin_example
30
#+end_example

Suppose x3 is changed and now x3(10) returns 40. I want to be notified that this change has happened (and get a diff of the two results).

PS: If this should be possible for visual (image) results, it would be really awesome.

PPS: Is there any testing framework that works like this? I.e., it only reports change, and does not need you to manually hardcode values?

Answer 1

One idea might be to save a copy of the current buffer, run the tests, and then run a diff command on them.

(defun run-tests ()
  (interactive)
  (copy-to-buffer "*my-test*" (point-min) (point-max))
  (org-babel-execute-buffer)
  (diff-buffers "*my-test*" (current-buffer)))

This will make a new buffer showing the diff (if any). There are lots of variations I suppose, like writing to files, different ways of diffing.

It would be hard I think to get this to work for images.

[edit]: here is a version that works on a copy of the buffer with ediff.

(defun run-tests ()
  (interactive)
  (let ((contents (buffer-string))
    (buf (current-buffer)))
    (with-current-buffer (get-buffer-create "*my-test*")
      (erase-buffer)
      (insert contents)
      (org-mode)
      (org-babel-execute-buffer)
      (ediff-buffers buf "*my-test*"))))