elpy error in process sentinel

Question

I've gone a long way to get emacs set up at my company. At home I'm using Emacs for many years on Linux. However, at work on Windows I'm struggling a bit getting it functioning smoothly.

When use Emacs to code in python via elpy I get the following error message:

error in process sentinel: elpy-rpc--default-error-callback: peculiar error: "exited abnormally with code 1"
error in process sentinel: peculiar error: "exited abnormally with code 1"

Googleing pointed me towards runing M-x elpy-rpc-reinstall-virtualenv. However, that doesn't work for me as I can't connect due to firewall issues. Installing packages (at work) works for me via conda and handing over a certificate (ZScaler) in the anaconda prompt shell. My workflow is:

1. create virtualenv via conda (with specific python version)
2. install python packages via Anaconda prompt shell into the virtualenv. Here I can add the certificate to be able to fetch data.
3. activate the virtualenv in Emacs using pyvenv

Not sure if the M-x elpy-rpc-reinstall-virtualenv would sovle my issue. I get asked to install automatically the RPC dependencies. However, if I hit y it fails to download as there is no certificate handed over (my assumption).

Elpy is installing the RPC dependencies...
Elpy failed to install some of the RPC dependencies, please use ‘elpy-config’ to install them.
WARNING: pip is configured with locations that require TLS/SSL, however the ssl module in Python is not available.
WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError("Can't connect to HTTPS URL because the SSL module is not available.")': /simple/jedi/
WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError("Can't connect to HTTPS URL because the SSL module is not available.")': /simple/jedi/
WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError("Can't connect to HTTPS URL because the SSL module is not available.")': /simple/jedi/
WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError("Can't connect to HTTPS URL because the SSL module is not available.")': /simple/jedi/
WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError("Can't connect to HTTPS URL because the SSL module is not available.")': /simple/jedi/
Could not fetch URL https://pypi.org/simple/jedi/: There was a problem confirming the ssl certificate: HTTPSConnectionPool(host='pypi.org', port=443): Max retries exceeded with url: /simple/jedi/ (Caused by SSLError("Can't connect to HTTPS URL because the SSL module is not available.")) - skipping
ERROR: Could not find a version that satisfies the requirement jedi (from versions: none)
ERROR: No matching distribution found for jedi
WARNING: pip is configured with locations that require TLS/SSL, however the ssl module in Python is not available.
Could not fetch URL https://pypi.org/simple/pip/: There was a problem confirming the ssl certificate: HTTPSConnectionPool(host='pypi.org', port=443): Max retries exceeded with url: /simple/pip/ (Caused by SSLError("Can't connect to HTTPS URL because the SSL module is not available.")) - skipping

I don't know how this can be done in Emacs? Anyone an idea how to fix this? With the process sentinel error it is not feasible to use Emacs as an IDE. It is getting super slow so any help would be much appreciated.

I'm running Emacs 28.1 at work

Update 1

Here is the output of wsl -l -v. Note my Windwos version number is higher than 18917. So I assume I'm still using WSL 1.

C:\Users>wsl -l -v

Copyright (c) Microsoft Corporation. All rights reserved.

Usage: wsl.exe [Argument]

Arguments:

    --install <Options>
        Install Windows Subsystem for Linux features. If no options are specified,
        the recommended features will be installed along with the default distribution.

        To view the default distribution as well as a list of other valid distributions,
        use 'wsl --list --online'.

        Options:
            --distribution, -d [Argument]
                Specifies the distribution to be downloaded and installed by name.

                Arguments:
                    A valid distribution name (not case sensitive).

                Examples:
                    wsl --install -d Ubuntu
                    wsl --install --distribution Debian

    --list, -l [Options]
        Lists distributions.

        Options:
            --online, -o
                Displays a list of available distributions for install with 'wsl --install'.

    --help
        Display usage information.

C:\Users>

Update 2

I've added to the pip.ini file the path to the certificate via pip config set global.cert "C:\Users\zscalarcert2.cer". However, running elpy-rpc-reinstall-virtualenv gives the error

Elpy is creating the RPC virtualenv (’c:/Users/ca/AppData/Roaming/.emacs.d/elpy/rpc-venv’)
Automatically install the RPC dependencies from PyPI (needed for completion, autoformatting and documentation) ? (y or n) y
Elpy is installing the RPC dependencies...
Elpy failed to install some of the RPC dependencies, please use ‘elpy-config’ to install them.
WARNING: pip is configured with locations that require TLS/SSL, however the ssl module in Python is not available.
WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError("Can't connect to HTTPS URL because the SSL module is not available.")': /simple/jedi/
WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError("Can't connect to HTTPS URL because the SSL module is not available.")': /simple/jedi/
WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError("Can't connect to HTTPS URL because the SSL module is not available.")': /simple/jedi/
WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError("Can't connect to HTTPS URL because the SSL module is not available.")': /simple/jedi/
WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError("Can't connect to HTTPS URL because the SSL module is not available.")': /simple/jedi/
Could not fetch URL https://pypi.org/simple/jedi/: There was a problem confirming the ssl certificate: HTTPSConnectionPool(host='pypi.org', port=443): Max retries exceeded with url: /simple/jedi/ (Caused by SSLError("Can't connect to HTTPS URL because the SSL module is not available.")) - skipping
ERROR: Could not find a version that satisfies the requirement jedi (from versions: none)
ERROR: No matching distribution found for jedi
WARNING: pip is configured with locations that require TLS/SSL, however the ssl module in Python is not available.
Could not fetch URL https://pypi.org/simple/pip/: There was a problem confirming the ssl certificate: HTTPSConnectionPool(host='pypi.org', port=443): Max retries exceeded with url: /simple/pip/ (Caused by SSLError("Can't connect to HTTPS URL because the SSL module is not available.")) - skipping

Done

Update 3

I'm using the same python version for the global and the virtualenv

C:\Users\ca>python --version
Python 3.9.0

After adding the global.cert I was able to install boto3 (a package I haven't had installed before) without handing over the --cert option:

C:\Users\ca>pip install boto3
Collecting boto3
  Using cached boto3-1.24.44-py3-none-any.whl (132 kB)
Requirement already satisfied: jmespath<2.0.0,>=0.7.1 in c:\program files\python39\lib\site-packages (from boto3) (1.0.1)
Requirement already satisfied: botocore<1.28.0,>=1.27.44 in c:\program files\python39\lib\site-packages (from boto3) (1.27.44)
Requirement already satisfied: s3transfer<0.7.0,>=0.6.0 in c:\program files\python39\lib\site-packages (from boto3) (0.6.0)
Requirement already satisfied: python-dateutil<3.0.0,>=2.1 in c:\program files\python39\lib\site-packages (from botocore<1.28.0,>=1.27.44->boto3) (2.8.2)
Requirement already satisfied: urllib3<1.27,>=1.25.4 in c:\program files\python39\lib\site-packages (from botocore<1.28.0,>=1.27.44->boto3) (1.26.11)
Requirement already satisfied: six>=1.5 in c:\program files\python39\lib\site-packages (from python-dateutil<3.0.0,>=2.1->botocore<1.28.0,>=1.27.44->boto3) (1.16.0)
Installing collected packages: boto3
Successfully installed boto3-1.24.44

It also worked in the virtualenv:

(srv_p) C:\Users\ca>pip install boto3
Collecting boto3
  Using cached boto3-1.24.44-py3-none-any.whl (132 kB)
Collecting s3transfer<0.7.0,>=0.6.0
  Using cached s3transfer-0.6.0-py3-none-any.whl (79 kB)
Collecting botocore<1.28.0,>=1.27.44
  Using cached botocore-1.27.44-py3-none-any.whl (9.0 MB)
Collecting jmespath<2.0.0,>=0.7.1
  Using cached jmespath-1.0.1-py3-none-any.whl (20 kB)
Collecting python-dateutil<3.0.0,>=2.1
  Using cached python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB)
Collecting urllib3<1.27,>=1.25.4
  Using cached urllib3-1.26.11-py2.py3-none-any.whl (139 kB)
Collecting six>=1.5
  Using cached six-1.16.0-py2.py3-none-any.whl (11 kB)
Installing collected packages: six, python-dateutil, jmespath, urllib3, botocore, s3transfer, boto3
Successfully installed boto3-1.24.44 botocore-1.27.44 jmespath-1.0.1 python-dateutil-2.8.2 s3transfer-0.6.0 six-1.16.0 urllib3-1.26.11

However, if I delete the global.cert option again, de-install and try to re-install boto3 it fails. That means the global.cert is picked up correctly:

C:\Users\ca>pip uninstall boto3
Found existing installation: boto3 1.24.44
Uninstalling boto3-1.24.44:
  Would remove:
    c:\program files\python39\lib\site-packages\boto3-1.24.44.dist-info\*
    c:\program files\python39\lib\site-packages\boto3\*
Proceed (y/n)? y
  Successfully uninstalled boto3-1.24.44

C:\Users\ca>pip install boto3
WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1122)'))': /simple/boto3/
WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1122)'))': /simple/boto3/
WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1122)'))': /simple/boto3/
WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1122)'))': /simple/boto3/
WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1122)'))': /simple/boto3/
Could not fetch URL https://pypi.org/simple/boto3/: There was a problem confirming the ssl certificate: HTTPSConnectionPool(host='pypi.org', port=443): Max retries exceeded with url: /simple/boto3/ (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1122)'))) - skipping
ERROR: Could not find a version that satisfies the requirement boto3 (from versions: none)
ERROR: No matching distribution found for boto3

Answer 1

I am fairly confident, based on reports issued to the Elpy github issues page as well as the circumstances that were communicated by OP in this question, that the problem coming up in this issue is a result of how virtual environments and pip interact within the elpy-rpc module. I will begin this answer by giving background on what I believe may be causing the problem within elpy-rpc, then offering some ways that were used to overcome the issue in OP's case.

---

Background

When elpy-rpc tries to install the dependencies it needs for a virtual environment, it does so with a command similar to py -m pip install [package list]. The problem as OP has observed is that it does not give a certificate to pip (which would normally be done with --cert path/to/cert. )

Moreover, there seem to be inconsistencies with how pip within elpy operates when it is called in a virtual environment versus globally. As OP (and I) have tested, when a global cert is given to pip config, it is used in any virtual environments. However, it does not appear to be the case when OP tested the same circumstances with elpy.

---

Workarounds

I will offer two workarounds: one based on what worked for OP in this question, and a way that I believe can work based on my reading of the source for elpy.

It appears that the following worked for OP in this question:

1. Add your certificate to the global config of pip. This answer offers a method for doing that in Windows. But in summary, these two commands are relevant:

pip config set global.cert path/to/ca-bundle.crt
pip config list

2. Reinstall elpy then try to set it up again.

I will offer the following method for anyone wishing to edit elpy-rpc to send the --cert option to the dependencies installation method.

1. Find where elpy-rpc.el is on your load path. You can find the source file in question on Github here.
2. Navigate to the function definition for elpy-rpc--install-dependencies. The entire function is available below:

(defun elpy-rpc--install-dependencies ()
  "Install the RPC dependencies in the current virtualenv."
  (if (y-or-n-p "Automatically install the RPC dependencies from PyPI (needed for completion, autoformatting and documentation) ? ")
      (with-temp-buffer
        (message "Elpy is installing the RPC dependencies...")
        (when (/= (apply 'call-process elpy-rpc-python-command
                         nil t nil
                         "-m" "pip" "install" "--upgrade"
                         (elpy-rpc--get-package-list))
                  0)
          (message "Elpy failed to install some of the RPC dependencies, please use `elpy-config' to install them.\n%s" (buffer-string))
          ))
    (message "Some of Elpy's functionnalities will not work, please use `elpy-config' to install the needed python dependencies.")))

3. On the indented line that starts with "-m" "pip" "install", add in two more strings: "--cert" and "path/to/cert/file". This will call the dependency installation command to pip with the cert option attached.