# Print process output line by line

Read process output line by line gives some partial ideas about how to deal with sub-process output in an intelligent manner, but no example and no working solution. I have a couple of examples, one of them works (badly) and the other doesn't.

To invoke the code: save the Emacs Lisp snippets as code1 and code2 then make them executable with chmod +x code* and run, e.g. ./code1 test.tex.

# test.tex

\documentclass{article}
\begin{document}
Emacs compilation test.
\end{document}


# this prints output from the inferior process all at once

#!/usr/bin/emacs --script

(call-process "pdflatex" nil '("*scratch*" t) nil (car argv))
(set-buffer (get-buffer "*scratch*"))

(prin1 (buffer-string))

(setq argv nil)

;;; Local Variables:
;;; mode:emacs-lisp
;;; End:


# This does not print output from the inferior process at all

#!/usr/bin/emacs --script

(defun my-process-filter (proc string)
(print "%s\n" string))

;; is that this would cause the output of the inferior process
;; to print out line by line.  It doesn't seem to work though.
(set-process-filter
(start-process "tex" "*Messages*" "pdflatex" (car argv))
#'my-process-filter)

(setq argv nil)

;;; Local Variables:
;;; mode:emacs-lisp
;;; End:


# tl;dr

Another "solution" that serves as an interactive "proof of concept" is to run M-x shell followed by pdflatex test.tex. Output from pdflatex is printed line by line. How to achieve the same result non-interactively?

• Start process is asynchronous; Emacs exits right after the process has started, without any chance to print anything. You have to wait for the process to exit. – lunaryorn Dec 19 '15 at 10:49
• Note though, that process filters don't work line-wise as far as I know. They receive arbitrary chunks of text, depending on Emacs' internal read buffer size. I'm not sure whether you can change that. – lunaryorn Dec 19 '15 at 10:51
• From the way you are using it, it seems like you want to read it in shell: why not use a pager? If you didn't know, pager is a program which, beside other things, is designed to buffer output from other programs which output to tty eg less and more. The way you'd use it is: myprogram | less – wvxvw Dec 20 '15 at 21:27
• @wvxvw, my aim is to use Emacs as a real-time "filter", closer to myprogram | tail -f. If process filters can't work line-wise, then the idea above will not work. However, there is certainly a solution existing somewhere "under the hood" since pdflatex can be run within an Emacs Shell and will print out text line-by-line there. I added this to the question as an illustrative tl;dr summary of the goal. – Joe Corneli Dec 21 '15 at 15:07
• @JoeCorneli As mentioned by lunaryorn, filter functions receive process output in chunks of arbitrary size. In order to process output line-wise, you can, for example, accumulate output in a temporary buffer until a whole line is detected, and only then operate on that line. See (elisp) Filter Functions for more information. – Basil Sep 24 '18 at 15:11

Emacs can invoke processes either synchronously or asynchronously. In the former case, the calling program has to wait for the process to terminate before processing the entire output in one go (whether character-wise, line-wise, or other-wise is up to the program). In the latter case, the program can still wait for the process to terminate before processing its output, but the program additionally has the ability to process intermediate output in chunks of any size it likes.

Here's an example of processing entire synchronous output in line-wise fashion:

(require 'cl-lib)                       ; For cl-oddp'

(defun my-process-line (line)
"Print LINE if it comprises an odd number."
(when (ignore-errors (cl-oddp (string-to-number line)))
(message ">>> %s" line)))

(mapc #'my-process-line (process-lines "seq" "10"))


This generates the following output:

>>> 1
>>> 3
>>> 5
>>> 7
>>> 9


If process-lines is too slow (less likely) or inflexible (more likely), you can achieve the same effect with call-process:

(mapc #'my-process-line
(split-string (with-output-to-string
(call-process "seq" nil standard-output nil "10"))
"\n" t))


Equivalently, with slightly better performance due to less GC:

(with-temp-buffer
(call-process "seq" nil t nil "10")
(goto-char (point-min))
(while (not (eobp))
(my-process-line
(buffer-substring-no-properties (point) (line-end-position)))
(forward-line)))


Here's the same example, but invoked asynchronously:

(defun my-sentinel (proc _msg)
"Process entire output of PROC line-wise."
(when (and (eq (process-status proc) 'exit)
(zerop (process-exit-status proc))
(buffer-live-p (process-buffer proc)))
(with-current-buffer (process-buffer proc)
(mapc #'my-process-line (split-string (buffer-string) "\n" t))
(kill-buffer))))

(let ((process-connection-type nil))
(set-process-sentinel (start-process "my-proc" " *my-proc*" "seq" "10")
#'my-sentinel))


Equivalently, using the Emacs 25 function make-process:

(make-process :name "my-proc"
:buffer " *my-proc*"
:command '("seq" "10")
:connection-type 'pipe
:sentinel #'my-sentinel)


Here's the same example, but with the output processed in chunks as it comes in, rather than after it's complete:

(defun my-filter (proc str)
"Process each line produced by PROC in STR."
(when (buffer-live-p (process-buffer proc))
(with-current-buffer (process-buffer proc)
(insert str)
(goto-char (point-min))
(while (progn (skip-chars-forward "^\n")
(not (eobp)))
(my-process-line (delete-and-extract-region (point-min) (point)))
(delete-char 1)))))

(let* ((process-connection-type nil)
(proc (start-process "my-proc" " *my-proc*" "seq" "10")))
(set-process-filter   proc #'my-filter)
(set-process-sentinel proc #'my-sentinel))


Equivalently, using make-process:

(make-process :name "my-proc"
:buffer " *my-proc*"
:command '("seq" "10")
:connection-type 'pipe
:filter #'my-filter
:sentinel #'my-sentinel)


As lunaryorn mentioned in a comment, the reason your sample asynchronous script fails to print anything is because it exits before being given the chance to accept output from its subprocess; you need to block somehow until the process has terminated. One way to do this is to use call-process et al. synchronously, but then you forgo the ability to process output lines as they come in. Another way is to block until the asynchronous subprocess is done:

(let* ((process-connection-type nil)
(proc (start-process "my-proc" " *my-proc*" "seq" "10")))
(set-process-filter   proc #'my-filter)
(set-process-sentinel proc #'my-sentinel)
(while (eq (process-status proc) 'run)
(accept-process-output proc)))


Equivalently, using make-process:

(let ((proc (make-process :name "my-proc"
:buffer " *my-proc*"
:command '("seq" "10")
:connection-type 'pipe
:filter #'my-filter
:sentinel #'my-sentinel)))
(while (eq (process-status proc) 'run)
(accept-process-output proc)))


The gory details for all of this and more are documented under (elisp) Processes.

*** Welcome to IELM ***  Type (describe-mode) for help.
ELISP> (mapc 'print
(split-string
(with-temp-buffer
(call-process "printf" nil t nil "%s\n" "one" "two" "three")
(buffer-substring-no-properties (point-min)
(point-max)))
"\n"))

"one"

"two"

"three"

""

("one" "two" "three" "")

ELISP>


This is all done in a M-x ielm buffer for demonstration. You can use this code anywhere Emacs Lisp code works in Emacs.

with-temp-buffer creates a temporary buffer. The expressions in its body are evaluated in sequence. The temporary buffer will hold the output of the process temporarily and be automatically killed when our Lisp code is done.

I chose to demonstrate with the external printf command but you can use any command you like to produce output lines.

call-process invokes the external printf command to print three words each followed by a newline. The output of the printf command is inserted into the temporary buffer.

buffer-substring-no-properties yields the full contents of the the temporary buffer. This is the return value of the with-temp-buffer form.

split-string breaks the string returned from with-temp-buffer on newline characters. It returns a list of strings with the newlines removed.

At this point we have a list of lines from the process output. You may process this list in any way Emacs Lisp can process lists. I chose to quote and print each output line separated by empty lines.

mapc applies the function from its first argument to the list in its second argument. Here I use the print function to be applied to the list. Here the list is being returned from the split-string function. mapc returns the original list unmodified.

print` inserts its output into the IELM buffer as a side effect. The output can be redirected to other buffers or processed in other ways, but that would make for an unnecessarily complex demonstration code.

• Could you add some text describing how this gives an answer? – Stefan Sep 24 '18 at 15:39
• I don't know what exactly you want to see. It does do what was asked for. I'll edit in an explanation. – user20119 Sep 26 '18 at 0:36
• I think @Stefan was asking less for a description of each Elisp construct used (which is what docstrings are for), and more for a justification of how your sample code answers the OP. Code-only and link-only answers leave readers guessing, particularly those less experienced with Emacs and Elisp. – Basil Sep 26 '18 at 8:54
• @user20119 The crux of your code-only answer seems to have been "just call the process synchronously, wait for its entire output to be received, and then split that into lines for line-wise processing". This may or may not have answered the OP, and it may or may not have even been what you meant, which is why adding some form of justification is important. – Basil Sep 26 '18 at 8:58
• @user20119 Re: "It does do what was asked for." Not necessarily or entirely, as it blocks until the whole process output has been received, whereas OP indicates they want to process each line as it's received. – Basil Sep 26 '18 at 9:00