If I run this in bash (it's a GraphViz command to make a picture of a graph based on the input file testgraph):

$ dot -Tpng testgraph.dot > graph.png

with this testgraph.dot:

digraph G {
    a -> b

I can execute it as many times as I want, and graph.png comes out right everytime. But if I execute it in eshell, the file will be corrupt if I don't move the old graph.png out of the way first. My first guess was that eshell must do appending by default rather than truncating, but if I test with echo and cat with plaintext it works as expected. If I compare one of the corrupt files against one of the regular ones, the broken ones are shorter:

$ ls -l graph*
-rw-r--r--   1 aaaaa        xxxx          3203 2015-12-21 15:24 graph.png
-rw-r--r--   1 aaaaa        xxxx          3087 2015-12-21 15:24 graph.png.broken

The file sizes are consistent, if I run the command fifty times without removing I will get the exact same size truncated file every time.

  • I can confirm. The last byte seems to be missing from the png generated from eshell. Using dot -Tsvg produces identical svg files in eshell and regular shell-mode, so maybe it's something to do with eshell mangling binary output? – Tyler Dec 21 '15 at 22:03
  • Additionnaly, eshell complains with 'Text is read only' if the file already exists, and open it in a buffer. – JeanPierre Dec 21 '15 at 22:22

Redirection in Eshell works by writing the output of the command to a temporary buffer that's visiting the target file, then saving the buffer to the target file.

By the way, “output” includes anything the command writes to the terminal. > in Eshell isn't the redirection of standard output (stdout) like it is in other shells, it redirects everything including standard error (stderr).

The output of a command is a sequence of bytes, but the content of a buffer is a sequence of characters. Normally, this doesn't matter: once the buffer is saved, it's again a series of bytes. Problems arise when there isn't a distinct character sequence for every byte sequence. If the conversion from characters to bytes is ambiguous (i.e. there are multiple byte sequences that are read as the same character), then the output may be a different byte sequence from the original; I don't know if this actually happens. If the conversion from bytes to characters is impossible, some bytes may be lost, and that happens in your case.

There are many ways to configure Emacs with respect to encodings and I'm not familiar with all of them. I'll report what I observed in Emacs 24.3 running on Linux, started with emacs -Q; other versions and especially user configuration, as well as the ambient locale, may cause different behavior.

With Emacs started in a UTF-8 locale:

/tmp $ printf 'a\xe3\n' >a; hd a

Emacs prompts me to select a coding system because \343 (\xe3) is not a valid byte sequence in UTF-8. It defaults to raw-text. I allow the default, and the file has the desired content.

00000000  61 e3 0a                                          |a..|

However, when the invalid byte is the last one in the file, it seems to be silently dropped.

/tmp $ printf 'a\xe3' >a; hd a
00000000  61                                                |a|

This smacks of a bug to me. I don't know if this is specific to Eshell or if it's an underlying issue with subprocess output.

If I start Emacs in the C locale, the behavior is different. The temporary buffer defaults to UTF-8, but the process's output is, somehow, interpreted as Latin-1. I can't explain this behavior.

/tmp $ printf 'a\xe3\n' >a; hd a
00000000  61 c3 a3 0a                                       |a...|
/tmp $ printf 'a\xe3' >a; hd a
00000000  61 c3 a3                                          |a..|

The output from dot may depend on its version. With mine, if I execute dot outside of Eshell, or if I run dot -Tpng testgraph.dot -o graph.png, I get a 3204-byte file that ends in

00000c80  ae 42 60 82                                       |.B`.|

But if I run dot -Tpng testgraph.dot >graph.png, the last byte is missing. As in my example above, the file ends with a partial UTF-8 byte sequence, which gets dropped by Eshell redirection.

I haven't been able to reproduce the 3087-byte file when running the command again, but I can imagine this happening if your Emacs configuration causes it to make some automatic encoding decision when saving the file the first time round, which then influences the way the subprocess output is read the second time round. With emacs-24.3 -Q, Emacs detects that the output is an image (because of the PNG magic header) and tells me in the Eshell buffer

/tmp $ dot -Tpng testgraph.dot >graph.png
/tmp $ dot -Tpng testgraph.dot >graph.png
Text is read-only

because the file is still open in Image mode.

All in all, output redirection in Eshell is not reliable for binary data or text that isn't encoded in your default encoding (for some value of “default encoding”), as of Emacs 24.3 (I also tested with 24.5, no improvement). This is a limitation in Eshell, arguably a bug.

| improve this answer | |
  • Damn, that might be the death knell for me being able to use eshell as a zsh replacement. Why doesn't eshell directly connect processes via pipes? I know that it wants to support things like redirect to a buffer, but it should be able to detect when the target is a buffer and only invoke special behavior in that case. – Joseph Garvin Dec 23 '15 at 14:56
  • @JosephGarvin I suspect the answer to why is because nobody wrote the code. I don't know if the author is accepting patches. I gave up replacing zsh long ago (and I don't even run it in term... Good old xterm. And emacsclient.). It's not “special behavior” though — eshell would have to either invoke a shell (which it doesn't do, and can't do easily since Unix has sh, Windows has cmd, and there might still be platforms running Emacs that have yet something else or nothing), or implement direct-to-file redirection itself (I'm not sure if Emacs even exposes the proper OS primitives to Lisp). – Gilles 'SO- stop being evil' Dec 23 '15 at 15:17
  • I wonder if this could be fixed by making eshell use the no-conversion encoding. (the pipes idea isn't possible with current Emacs, afaik) – npostavs Feb 1 '17 at 5:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.