Tamil characters show as garbage in shell buffer but ok in files

Question

When printing non-latin characters like the ones from my mother tongue Tamil in shell, it appears weird(left side window in the image). When I open the file with Tamil characters they are rendered correctly(right side window in the image). I have set the shell locale to utf-8 but the problem still persists.

vanangamudi@kaikuttai:~/home/projects/code/oneoff-experiments/tamil-etymdict-visualizer$ python3 -i generate-dot.py
100% 53753/53753 [00:00<00:00, 163166.71it/s]
Traceback (most recent call last):
  File "generate-dot.py", line 32, in <module>
    write_dot(G, 'graph.dot')         
  File "/home/vanangamudi/.local/lib/python3.6/site-packages/networkx/drawing/nx_agraph.py", line 192, in write_dot
    A = to_agraph(G)
  File "/home/vanangamudi/.local/lib/python3.6/site-packages/networkx/drawing/nx_agraph.py", line 151, in to_agraph
    A.add_node(n)
  File "/home/vanangamudi/.local/lib/python3.6/site-packages/pygraphviz/agraph.py", line 309, in add_node
    node = Node(self, nh=nh)
  File "/home/vanangamudi/.local/lib/python3.6/site-packages/pygraphviz/agraph.py", line 1611, in __new__
    n = super(Node, self).__new__(self, gv.agnameof(nh), graph.encoding)
TypeError: decoding to str: need a bytes-like object, NoneType found
>>> G.nodes
NodeView(('நà¯à®°à¯à®à¯à®à¯à®´à®¿|நà¯à®°à¯à®à¯à®à¯à®´à®¿|பà¯', '(n', 'நà¯à®°à¯', 'பà¯à®´à¯', 'தà¯à®³à¯à®¤à¯à®¤à®²à¯', 'à®à¯à®±à¯à®¤à®²à¯', 'fowl', 'à®à¯à®´à®¿', 'வனà¯à®©à®¤à¯à®²à®¿à®à¯|வனà¯à®©à®¤à¯à®²à®¿à®à¯|பà¯', '|பà¯', 'à®à®³à®µà®¾à®à®®à¯|à®à®³à®µà®¾à®à®®à¯|பà¯', 'à®à¯', 'à®à¯', 'வி', 'à®à¯à®à¯à®¤à¯à®¤à¯à®£à®¿|à®à¯à®à¯à®¤à¯à®¤à¯à®£à®¿|பà¯', 'à®à¯à®©à¯à®±à®¾à®µà®¿', 'à®à®²à¯à®à¯|à®à®²à¯à®à¯

Adding locale information as pointed out by @Rusi,

vanangamudi@kaikuttai:~$ locale
LANG=en_IN
LANGUAGE=en_IN:en
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=en_US.UTF-8
vanangamudi@kaikuttai:~$ locale -a
C
C.UTF-8
en_AG
en_AG.utf8
en_AU.utf8
en_BW.utf8
en_CA.utf8
en_DK.utf8
en_GB.utf8
en_HK.utf8
en_IE.utf8
en_IL
en_IL.utf8
en_IN
en_IN.utf8
en_NG
en_NG.utf8
en_NZ.utf8
en_PH.utf8
en_SG.utf8
en_US.utf8
en_ZA.utf8
en_ZM
en_ZM.utf8
en_ZW.utf8
POSIX
ta_IN
ta_IN.utf8
ta_LK
ta_LK.utf8
vanangamudi@kaikuttai:~$ 

Answer 1

Diagnosis

Clearly at this point we have only insufficient data. Anyways... here's my tentative diagnosis

• LANG is set en_IN (no utf8) in /etc/default/locale; LANGUAGE too is set there to en_IN:en
• LC_ALL is set somewhere else to en_US.utf8
• In short, your system is in a tug-o-war between IN-dia and US-a, between utf8 and legacy encoding and emacs is getting confused.

Suggested Treatment

Assuming the above somewhat true, you need to

• Add the utf8 to LANG setting
• Find and remove wherever LC_ALL is being set
• verify working by calling locale. All should be same. LC_ALL should show as unset
• Try emacs without any attempted locale related customizations. I would guess that shell-inside-emacs has more things that could go wrong than simply reading/writing Tamil files. And python-unicode even more so. So for testing purposes I'd start with the reading/writing Tamil files. For python-unicode, first try in a shell outside emacs.

Finding LANG/LC_* settings

This can be a non-trivial treasure hunt. Some likely culprits

• /etc/default/locale (debian/ubuntu)
• /etc/environment (older debian/ubuntu)
• /etc/profile
• In your $HOME
  • .pam_environment
  • .profile
  • .bash_profile
  • .bashrc
• etc...

Unfortunately this list cannot be exhaustive😅 And at this point is getting quite far removed from emacs.

---

Also please report

 M-: (process-coding-system (get-process "???"))

where ??? Is shell or Shell or *shell* ... I'm not exactly sure which will work

And, if necessary, use set-process-coding-system. ( manual )