Under Windows, Emacs cannot handle encodings correctly for subprocesses, can it?

Question

I have two simple text files on my D: drive, as shown below.

D:\我.txt

I

D:\你.txt

You

After I execute M-x diff and type the correct filenames (Yes, I can input Chinese and filename autocomplete works fine for Chinese), the output give the following.

diff -u "d:/我.txt" "d:/你.txt"
diff: "d:/我.txt": No such file or directory
diff: "d:/ä½ .txt": No such file or directory

Diff finished (diff error).  Wed Jan  4 02:13:20 2017

I have MSYS2 installed with diff residing on D:\msys32\usr\bin\diff.exe.

I run Emacs 25.1.1 on Windows 7 (en-US), so the w32-unicode-filenames is set to t, which means the native Windows Unicode APIs is used when passing file names to the OS. This is good, Unicode file names display correctly in dired-mode, and autocomplete for Unicode file names also works seamlessly. So I suspect that Unicode file names only cause problem when creating subprocesses with Unicode arguments.

To verify that, I execute M-x shell and do the following.

Microsoft Windows [Version 6.1.7601]
Copyright (c) 2009 Microsoft Corporation.  All rights reserved.

$ cd
cd
d:\

$ type 我.txt
type 我.txt
The system cannot find the file specified.

$ type 你.txt
type 你.txt
The system cannot find the file specified.
Error occurred while processing: \344\275.
The system cannot find the file specified.
Error occurred while processing: .txt.

$ where cat
where cat
D:\msys32\usr\bin\cat.exe

$ cat 我.txt
cat 我.txt
cat: ''$'\303\246\313\206\342\200\230''.txt': No such file or directory

$ cat 你.txt
cat 你.txt
cat: ''$'\303\244\302\275\302\240''.txt': No such file or directory

Autocomplete for Unicode file name also ceases to work in shell-mode.

And I also try similar commands in eshell-mode invoked by M-x eshell.

Welcome to the Emacs shell

d:/ $ which which
eshell/which is a compiled Lisp function in ‘esh-cmd.el’
d:/ $ which cat
eshell/cat is a compiled Lisp function in ‘em-unix.el’
d:/ $ cat 我.txt
I
d:/ $ cat 你.txt
You
d:/ $ which ls
eshell/ls is a compiled Lisp function in ‘em-ls.el’
d:/ $ ls 我.txt 
我.txt
d:/ $ ls 你.txt 
你.txt
d:/ $ which ls.exe
d:/msys32/usr/bin/ls.exe
d:/ $ ls.exe 我.txt 
/usr/bin/ls: cannot access ''$'\303\246\313\206\342\200\230''.txt': No such file or directory
d:/ $ ls.exe 你.txt 
/usr/bin/ls: cannot access ''$'\303\244\302\275\302\240''.txt': No such file or directory

It almost confirms to me the issue is about the subprocess encoding. Calling internal Elisp functions with Unicode arguments works fine but calling external executables with Unicode arguments doesn't work. I also experiment with various encoding for subprocess. I set them in my init file like this.

(setq default-process-coding-system '(raw-text-dos . raw-text-unix))

But to no avail, none of my attempts work.

Answer 1

This is not an Emacs issue per se. It has to do with the command interpreter default code page. You need to change the "shell" code page to utf8 with chcp 65001. By default it sets the code page to Latin-1 or CP850 (which I believe are the same).

Updated answer after doing some more research

It turns out Emacs has a history of problems with Chinese fonts. There's a mode called chinese-fonts-setup here I installed it and that fixed the problem in my machine (I'm running GNU Emacs 24.5.1 (i686-pc-mingw32) on Windows 8.1). BTW install instructions are in Chinese, had to Google Translate + guess