Asynchronously wait for output from a comint process

Question

First of all, a disclaimer. I have researched this many times, and I am pretty sure that I have already found the answer one way or another, but I just don't understand it.

My problem is the following:

• I have a process running through comint
• I want to send a line of input, capture output and see when it is over (when the last line of the output matches the regexp for a prompt)
• only when the process has finished sending output, I want to send another line of input (for example).

For a bit of background, think of a major mode implementing interaction with a program, which may return an arbitrary amount of output, in arbitrarily long time. This should not be an unusual situation, right? Okay, maybe the part where I need to wait between inputs is unusual, but it has a few advantages over sending the input as a whole:

• the output buffer is nicely formatted: input output input output...
• more importantly, when sending a lot of text to a process, the text is cut into pieces and the pieces are pasted back by the process; the cutting points are arbitrary and this sometimes makes invalid input (my process won't paste back correctly an input cut in the middle of an identifier, for example)

Anyway, unusual or not, it turns out that it is complicated. Right now, I am using something along the lines of

(defun mymode--wait-for-output ()
  (let ((buffer (mymode-get-buffer)))
    (with-current-buffer buffer
      (goto-char (point-max))
      (forward-line 0)
      (while (not (mymode-looking-at-prompt))
        (accept-process-output nil 0.001)
        (redisplay)
        (goto-char (point-max))
        (forward-line 0))
      (end-of-line))))

and I call this every time after sending an input line, and before sending the next one. Well... it works, that's already something.

But it also makes emacs hang while waiting for the output. The reason is obvious, and I figured that if I included some kind of asynchronous sleep-for (for example 1s) in the loop it would delay the output by 1s, but suppress the hanging. Except that it seems that this kind of asynchronous sleep-for doesn't exist.

Or does it? More generally, is there an idiomatic way of achieving this with emacs? In other words:

How to send input to a process, wait for the output, then send more input, asynchronously?

When searching around (see the related questions), I have mainly seen mentions of sentinels (but I don't think it applies in my case, since the process doesn't finish) and of some comint hooks (but then what? should I make the hook buffer-local, turn my "evaluate the remaining lines" into a function, add this function to the hook and clean the hook afterwards? that sounds really dirty, doesn't it?).

I am sorry if am not making myself clear, or if there is really an obvious answer available somewhere, I am really confused by all the intricaties of process interaction.

If needed, I can make this all a working example, but I am afraid that it would just make one more "specific process question with specific process answer" like all those I found earlier and didn't help me.

Some related questions on SO:

• https://stackoverflow.com/q/16815598/1083706
• https://stackoverflow.com/q/3572532/1083706
• https://stackoverflow.com/q/6578373/1083706

Answer 1

First of all, you shouldn't be using accept-process-output if you want asynchronous processing. Emacs will accept output every time when it is waiting for user input.

The proper way to go is to use filter functions to intercept the output. You don't need to create or delete the filter(s) depending on whether you still have lines to send. Rather, you will usually declare a single filter for the lifetime of the process and use buffer-local variables to track the state and do different things as needed.

The low-level interface

Filter functions are what you're looking for. Filter functions are to output what sentinels are to termination.

(defun mymode--output-filter (process string)
  (let ((buffer (process-buffer process)))
    (when (buffer-live-p buffer)
      (with-current-buffer buffer
        (goto-char (point-max))
        (forward-line 0)
        (when (mymode-looking-at-prompt)
          (do-something)
          (goto-char (point-max)))))))

Look at the manual or at many examples that come with Emacs (grep for process-filter in the .el files).

Register your filter function with

(set-process-filter 'mymode--output-filter)

The comint interface

Comint defines a filter function which does a few things:

• Switch to the buffer that should contain the process output.
• Run the functions in the list comint-preoutput-filter-functions, passing them the new text as an argument.
• Perform some ad hoc duplicate prompt elimination, based on comint-prompt-regexp.
• Insert the process output at the end of the buffer
• Run the functions in the list comint-output-filter-functions, passing them the new text as an argument.

Given that your mode is based on comint, you should register your filter in comint-output-filter-functions. You should set comint-prompt-regexp to match your prompt. I don't think Comint has a built-in facility to detect a complete output chunk (i.e. between two prompts), but it can help. The marker comint-last-input-end is set at the end of the last input chunk. You have a new output chunk when the end of the last prompt is after comint-last-input-end. How to find the end of the last prompt depends on the version of Emacs:

• Up to 24.3, the overlay comint-last-prompt-overlay spans the last prompt.
• Since 24.4, the variable comint-last-prompt contains markers to the start and end of the last prompt.

(defun mymode--comint-output-filter (string)
  (let ((start (marker-position comint-last-input-end))
        (end (if (boundp 'comint-last-prompt-overlay)
                 (and comint-last-prompt-overlay (overlay-start comint-last-prompt-overlay))
               (and comint-last-prompt (cdr comint-last-prompt))))
  (when (and start end (< start end))
    (let ((new-output-chunk (buffer-substring-no-properties start end)))
      ...)))

You may want to add protections in case the process emits output in a sequence other than {receive input, emit output, display prompt}.