13

Is there a good general customization of org-babel to run asynchronously? Recently I plan to use MATLAB via org-babel, but I would like it in an async manner, since some calculations do take time.

I do not wish to customize ob-matlab only. This is because I think it should be done in the level of framework instead of an application. In other words, a same modification should enable async feature for other language extensions e.g. R language.

Does anyone have a good solution? So far I have tried async.el as well as deferred.el to modify org-babel-execute-safely-maybe that can be found in ob-core.el at the moment.

  • Another hint is can pass babel block to screen or tmux. – stardiviner Apr 30 '16 at 3:08
  • I've never implemented that, but it sounds possible. Thanks. – diadochos May 1 '16 at 7:30
  • I guess I am accepting my own answer since there hasn't been any other solution posted for the last one month. – diadochos Jun 16 '16 at 15:15
6

I have so far discovered that spawning a new Emacs process is a solution.

Here is what I have done.

1. Add a function to start an external emacs process.

init.el

(defvar my/async-emacs-repl-org-babel-init-file "~/.emacs.d/org-babel-async-init" "File to load on executing async babel evaluation.")

(defun my/async-emacs-repl--start (process-name init-file)
  "Start a new Emacs process as a REPL server."
  (async-shell-command (concat
                        "TERM=vt200 emacs --batch -nw"
                        " --eval '(load \"" init-file "\")'"
                        " --eval '(while t (print (eval (read))))'"
                        )
                       process-name))

(defun my/async-emacs-repl--org-babel--start-server ()
  "Starts an Emacs process for async org-babel execution."
  (my/async-emacs-repl--start "*org-babel-async*" my/async-emacs-repl-org-babel-init-file))

(defun my/async-emacs-repl--org-babel--start-if-not-exists ()
  "Starts an Emacs process if the process does not exist."
  (if (not (get-buffer-process "*org-babel-async*")) (my/async-emacs-repl--org-babel--start-server)))

(defun my/async-emacs-repl--org-babel--execute--build-command (file-name line-number)
  "Build the command for executing `org-babel-execute-src-block'."
  (concat
   "(progn"
   " (find-file \"" file-name "\")"
   " (revert-buffer t t)"
   " (goto-line " (number-to-string line-number) ")"
   " (org-babel-execute-src-block t)"
   " (save-buffer)"
   ")"
   "\n"))

(defun my/async-emacs-repl--org-babel--execute (process-name file-name line-number)
  "Sends the command to the server to run the code-block the cursor is at."
  (process-send-string
   process-name
   (my/async-emacs-repl--org-babel--execute--build-command file-name line-number)))

(defun my/async-emacs-repl-org-babel-do-execute ()
  "Run org babel execution at point."
  (my/async-emacs-repl--org-babel--execute "*org-babel-async*" (buffer-file-name) (line-number-at-pos)))

(defun my/async-emacs-repl-org-babel-execute ()
  "Run by the user. Executes command. Starts buffer if not exists."
  (interactive)
  (save-buffer)
  (my/async-emacs-repl--org-babel--start-if-not-exists)
  (my/async-emacs-repl-org-babel-do-execute))

2. Add a config file to load in the new emacs process.

The function above starts emacs in the --batch mode. Thus the normal init.el will not be loaded.

Instead, we want to create a shorter configuration file (to load paths and so on).

The path to our new config file is stored in async-emacs-repl-org-babel-init-file in the snippet above.

org-babel-async-init.el

;; 1
(package-initialize)

;; 2
(setq org-confirm-babel-evaluate nil)

;; 3
(let ((my/org-babel-evaluated-languages
       '(emacs-lisp
         ditaa
         python
         ruby
         C
         matlab
         clojure
         sh
         dot
         plantuml)))
  (org-babel-do-load-languages
   'org-babel-load-languages
   (mapcar (lambda (lang)
             (cons lang t))
           my/org-babel-evaluated-languages)))

Here we ...

  1. Add package paths.
  2. Tell org-mode to not ask whether to execute code block.
  3. Tell org-babel which languages are necessary.

Footnote 1: Without this setting, the evaluation will fail with "No org-babel-execute function for $lang!"

Footnote 2: Of course you can load the normal init.el instead of creating a new config file, if you wish. Do that by adding (setq org-babel-async-init-file "~/.emacs.d/init") to your init.el. But I think creating a configuration file for this task is more straightforward.

3. Additionally...

Add to init.el

;; This will stop the new process buffer from getting focus.
(setq display-buffer-alist (append display-buffer-alist '(("*org-babel-async*" display-buffer-no-window))))

;; This will automatically show the result section.
(global-auto-revert-mode 1)

Add to org-babel-async-init.el

;; This will skip the "Save anyway?" confirmation of automatically saving the file when you also edited the buffer from Emacs while an asynchronous process is running.
(defun advice:verify-visited-file-modtime (orig-func &rest args) t)
(advice-add 'verify-visited-file-modtime :around 'advice:verify-visited-file-modtime)

;; This will skip the "Select coding system" prompt that appears when the result is inserted. This may vary among environments.
(setq coding-system-for-write 'utf-8)

;; This will skip the "changed on disk; really edit the buffer?" checking.
(defun ask-user-about-supersession-threat (fn) "blatantly ignore files that changed on disk")

Add to org-babel-async-init.el (you may not need these. These are for MATLAB)

;; This will set MATLAB cli path.
(setq-default matlab-shell-command "/Applications/MATLAB_R2016a.app/bin/matlab")
;; The MATLAB cli path can be obtained by running `fullfile(matlabroot, 'bin')` in your MATLAB.

;; This will stop MATLAB from showing the splash (the MATLAB logo) at the beginning.
(setq-default matlab-shell-command-switches '("-nodesktop" "-nosplash"))

Add to org-babel-async-init.el (you may not need these. These are for Julia, R and other languages that use ESS.)

;; This will enable :session header in Julia and other languages that use ESS (Emacs speaks statistics).
(load "/path/to/ess-site")
;; This will suppress ESS from prompting for session directory.
(setq ess-ask-for-ess-directory nil)

4. Usage

(After the setup above.)

  1. Move cursor to the code snippet you want to execute.
  2. Run M-x my/async-emacs-repl-org-babel-execute (instead of doing C-c C-c). This will start an external Emacs process as a REPL server if needed, and then execute the source block you are at.

Acknowledgments

I have learned the idea of starting an emacs process for org-babel evaluation from this post. I would like to thank the author.

Comments for customization

The idea here is simple. Start a new emacs process as a REPL for Elisp, do find-file to the same .org file we are editing, goto-line to the same cursor point, run org-babel-execute-src-block, save-buffer. Stop exiting until the user stops the process (Otherwise, graphs would disappear immediately after they are shown). One can naturally think about extending this by:

  • Using org-mode's C-c C-c instead of running functions by hand / setting a new keybind (which can be achieved by advices).
  • Conditionally switching process name according to :session variable and the language
  • Conditionally switching init files based on the language.

In fact, the success of this approach seems to me to be showing a general way of developing async features in Emacs. Creating a "commands" layer, add scripts to do tasks, and have a framework for starting and reusing emacs processes. Just like Symfony framework of PHP (PHP doesn't have threads) has Command features.

Edit history

Refactored code (2016-04-02). Solution now reuses an Emacs process (2016-04-02). Solution now simplified and has only one interactive command to run (2016-04-02. Added configuration (2016-04-12).

  • Have you seen async.el? – PythonNut Apr 12 '16 at 14:11
  • Yes, I have. It essentially starts a new process of Emacs, and runs the lambda function given to it. I didn't use it for this solution because I couldn't find a way to send data to the new process. Communicating the process is necessary if you want to use the :session feature of org-babel. – diadochos Apr 13 '16 at 15:37
  • Thanks for working on this solution. I tried it but I get this error message: TERM=vt200 emacs --batch -nw --eval '(load "~/.emacs.d/org-babel-async-init")' --eval '(while t (print (eval (read))))': exited abnormally with code 255. Sorry, this should be a comment and not an answer, but I just don't have enough points. – mhartm Apr 25 '16 at 5:21
  • After executing that, do you see a buffer named "org-babel-async"? If you can find one, that buffer probably contains more information about the error. "exited abnormally with code 255" generally occurs when the program you wanted to run on the spawned emacs process failed. Possible ways out: 1) Check if you have the file specified in my/async-emacs-repl-org-babel-init-file. If you don't, create one as described above. 2) Check if you have listed the language you want to use in org-babel-do-load-languages. 3) The #+SRC_BEGIN block you are executing contains a bug. – diadochos Apr 27 '16 at 15:37
  • Okay, so the issue was that I need to save my org file before running M-x my/async-emacs-repl-org-babel-execute, otherwise the "org-babel-async" buffer will complain: ...t/Dropbox/org/work.org locked by maarhart@htkl... (pid 68694): (s, q, p, ?)? Please type q, s, or p; or ? for help. So if this can be solved, it would be fantastic. Thanks anyway for this, it is amazing! By the way, is it possible to bind it to C-c C-c or will it conflict with org-mode? – mhartm Apr 28 '16 at 5:44
1

For future references and out-of-box libraries. I will accept this one for the answer because these are more recent.

ob-async

Here's a library using async.el https://github.com/linktohack/ob-async

org-babel-eval-in-repl

My other solution (available in melpa): https://github.com/diadochos/org-babel-eval-in-repl

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.