I have a python file which defines a function and calls it.

def remove_whitespace(input_file, output_file):
    with open(input_file) as f:
        # do something

If I execute this file in Emacs (version 24.5.1), I recieve the error:

UnicodeEncodeError: 'ascii' codec can't encode character '\u2013' in position 1798: ordinal not in range(128)

I followed the error message and found many known issues relating to encoding. I altered my function to try filtering out anything outside of the ascii range 0-128 (leaving standard-ish only characters). I got the same error messages. So python is unable to read the file in the first place it seems. I should've recognised that sooner.

I then used the codecs module to open the file:

import codecs
with codecs.open(my_file, encoding="utf-8") as f:
    # do something

... but still received the same error.

Just out of curiousity, I exectued the file from terminal with the following:

python3 remove_whitespace.py

which worked! No errors, the output file was as expected.

I looked into elpy's congig with M-x elpy-config and saw that the shell-interpreter-arguments contains an additional -i flag. This simply puts the shell into interactive mode after executing the script, so I assume cannot affect anything.

I then preformed one last test - executing the file from an Emacs terminal: eshell. This gave me the same error as before, which again points the blame to Emacs itself (and less so at elpy!)

Can anybody shed light onto this Emacs/Elpy issue? Where is Emacs assuming/enforcing some encoding? I am not printing those non-ascii characters within my function, so didn't imagine Emacs even tries to interpret them itself.


A solution that I found over on Stackoverflow was the following proposed here for a UnicodeDecodingError, which did work for me:

(setenv "LANG" "en_US.UTF-8")
(setenv "LC_ALL" "en_US.UTF-8")
(setenv "LC_CTYPE" "en_US.UTF-8")

I only actually required the first of the three lines in my case.

Further suggestions specific to Python are in this post, namely to add this to your .emacs init file:

(setenv "PYTHONIOENCODING" "utf-8")

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.