UnicodeEncodeError executing python in Emacs - not in Terminal

Question

I have a python file which defines a function and calls it.

def remove_whitespace(input_file, output_file):
    with open(input_file) as f:
        f.readlines()
        # do something

If I execute this file in Emacs (version 24.5.1), I recieve the error:

UnicodeEncodeError: 'ascii' codec can't encode character '\u2013' in position 1798: ordinal not in range(128)

I followed the error message and found many known issues relating to encoding. I altered my function to try filtering out anything outside of the ascii range 0-128 (leaving standard-ish only characters). I got the same error messages. So python is unable to read the file in the first place it seems. I should've recognised that sooner.

I then used the codecs module to open the file:

import codecs
with codecs.open(my_file, encoding="utf-8") as f:
    # do something

... but still received the same error.

Just out of curiousity, I exectued the file from terminal with the following:

python3 remove_whitespace.py

which worked! No errors, the output file was as expected.

I looked into elpy's congig with M-x elpy-config and saw that the shell-interpreter-arguments contains an additional -i flag. This simply puts the shell into interactive mode after executing the script, so I assume cannot affect anything.

I then preformed one last test - executing the file from an Emacs terminal: eshell. This gave me the same error as before, which again points the blame to Emacs itself (and less so at elpy!)

Can anybody shed light onto this Emacs/Elpy issue? Where is Emacs assuming/enforcing some encoding? I am not printing those non-ascii characters within my function, so didn't imagine Emacs even tries to interpret them itself.

Answer 1

A solution that I found over on Stackoverflow was the following proposed here for a UnicodeDecodingError, which did work for me:

(setenv "LANG" "en_US.UTF-8")
(setenv "LC_ALL" "en_US.UTF-8")
(setenv "LC_CTYPE" "en_US.UTF-8")

I only actually required the first of the three lines in my case.

Further suggestions specific to Python are in this post, namely to add this to your .emacs init file:

(setenv "PYTHONIOENCODING" "utf-8")