Shell IO Behavior For Emacs' Shells

Question

The GNU page for Eshell says that

Eshell is not a replacement for system shells such as bash or zsh. Use Eshell when you want to move text between Emacs and external processes; if you only want to pipe output from one external process to another (and then another, and so on), use a system shell, because Emacs's IO system is buffer oriented, not stream oriented, and is very inefficient at such tasks. If you want to write shell scripts in Eshell, don't; either write an elisp library or use a system shell.

This makes it sound as if this IO system being buffer oriented was unique to Eshell. I'm new to Emacs and still trying to wrap my head around what the various shell modes do (and what the differences between them are), but unless I'm quite mistaken, M-x shell, ansi-term, multi-term, etc. all use this form of buffer oriented IO too, correct? So this criticism isn't something unique to Eshell -- it is a valid criticism for running shells in Emacs in general?

In other words, zsh, say, in Emacs (through M-x shell or multi-term) would not inherently be superior to Eshell due to this consideration?

If this is the case, then I suppose if one wants to "main" Eshell it is a good idea to keep a real terminal emulator running for big compile commands, using cat on files that might end up being super long, and other things that are stream oriented. Is this what people who us Eshell/M-x shell almost exclusively do to avoid hanging Emacs with lots of output?

Answer 1

M-x shell, ansi-term, multi-term, etc. all use this form of buffer oriented IO too, correct? So this criticism isn't something unique to Eshell -- it is a valid criticism for running shells in Emacs in general?

M-x shelland M-x ansi-term (never heard of multi-term) all spawn process that you interact with directly. The commands you type into your buffer get sent to the spawned processes' standard input. Command pipelines you enter will use pipes (pipe(2) or similar, I'd imagine) to pass data to each other.

Running M-x eshell does not spawn an external process, it runs within Emacs and uses its IO subsystem. Command pipelines you enter are subject to its internals which, according to the docs, includes buffering.

For example, in an M-x shell:

bash-3.2$ ls -l x.wsdl 
-rw-r--r--  1 sshaw  wheel  236658 Jan 21 01:56 x.wsdl
bash-3.2$ time cat x.wsdl | wc -l
    3786

real    0m0.004s
user    0m0.002s
sys     0m0.004s

And, in an Eshell:

/tmp $ time cat /tmp/x.wsdl  | wc -l
    1356
2.224 secs

Now, is that due to Eshell buffering x.wsdl? Not sure. Many of the popular shell commands are re-implemented too, so executing cat, ls, etc... run elisp versions.

Two big benefits to using Eshell are that it allows one to execute Emacs' functions, you can even use them in pipelines:

/tmp $ setq foo 10
10
/tmp $ * $foo 10
100
/tmp $ buffer-string | grep setq
~ $ setq foo 10

(You can also use parenthesis, i.e., (setq foo 10), (buffer-string) | grep setq)

For more info see:

• Interactive Subshell
• Introduction to Eshell video
• Mastering Eshell

… it is a good idea to keep a real terminal emulator running for big compile commands, using cat on files that might end up being super long, and other things that are stream oriented. Is this what people who us Eshell/M-x shell almost exclusively do to avoid hanging Emacs with lots of output?

Not sure, I don't use Eshell often but, from what I read, it sounds like a lot of folks use Eshell for a large percentage of their shell processing. Of course, you see what the limits are above so you'd have to adjust your workflow as necessary.