I remember @SNowick_WMF also experienced this when she first started using SWAP, and she may still be using the ~/venv/bin/pip workaround. Shay, what server have you been using?

I just logged into stat1004 to check out its new JupyterHub installation.

Did you do this via JupyterHub in the browser, or just in the terminal via ssh? If the latter, you need to activate ~/venv like you would any normal virtualenv (source ~/venv/bin/activate); it is only automatically activated for you in your Jupyter notebook server.

In T247752#5972405, @Ottomata wrote:

Did you do this via JupyterHub in the browser, or just in the terminal via ssh? If the latter, you need to activate ~/venv like you would any normal virtualenv (source ~/venv/bin/activate); it is only automatically activated for you in your Jupyter notebook server.

This was using JupyterHub's in-browser terminal, so it should have worked automatically. I helped Shay troubleshoot back when I encountered this, and I know she was in the JupyterHub terminal as well.

@nshahquinn-wmf if you have a Jupyter terminal handy, what does the following command say which pip3? Where it works as your expect I mean (I guess on notebook1003/4)

In T247752#5972722, @elukey wrote:

@nshahquinn-wmf if you have a Jupyter terminal handy, what does the following command say which pip3? Where it works as your expect I mean (I guess on notebook1003/4)

Sure!

neilpquinn-wmf@notebook1003:~$ which pip3
/home/neilpquinn-wmf/venv/bin/pip3

In T247752#5971999, @nshahquinn-wmf wrote:

I remember @SNowick_WMF also experienced this when she first started using SWAP, and she may still be using the ~/venv/bin/pip workaround. Shay, what server have you been using?

@SNowick_WMF encountered this on notebook1004, so it seems to have to do with new Jupyter environments rather than specific servers.

I just have export PATH=/home/bearloga/venv/bin:${PATH} in my ~.bashrc file on all the machines.

@elukey @Ottomata: perhaps that sort of thing can be done automatically for users as part of the initial homedir setup??

I use notebook1004, to fix this I added aliases in my .bash_profile and
haven't had any issues since.

alias py='/home/snowick/venv/bin/python3'
alias pip='/home/snowick/venv/bin/pip'
alias jupyter='/home/snowick/venv/bin/jupyter'

I think that Neil is right, it is probably something that I missed when upgrading SystemdSpawner on all the notebooks/stats. For example, on notebook1003:

elukey@notebook1003:~$ echo $PATH
/home/elukey/venv/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

At the beginning of the PATH there is the venv's bin, that should come from this directive in Jupyterhub:

c.SystemdSpawner.extra_paths = ['/home/{USERNAME}/venv/bin']

On notebook1003 works, on the rest of the node it doesn't. This is probably a PEBCAK from my side, will try to find what I missed (apologies).

Ok so the $PATH is correctly picked up in kernels (I tested a py3 one) but not from the Terminal. Not sure what different on notebook1003, it has been upgraded as well like 1004.

For some mysterious reasons, I can now see PATH properly populated in notebook1004:

elukey@notebook1004:~$ echo $PATH
/home/elukey/venv/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

I have restarted jupyterhub and also my own notebook. @nshahquinn-wmf would you mind to try on 1004 and see if it works? (restart your notebook I mean)

In T247752#5975620, @elukey wrote:

Ok so the $PATH is correctly picked up in kernels (I tested a py3 one) but not from the Terminal. Not sure what different on notebook1003, it has been upgraded as well like 1004.

Ah, yeah, I just tried in a Python notebook on stat1004, and I was able to use pip without problems.

In T247752#5975777, @elukey wrote:

@nshahquinn-wmf would you mind to try on 1004 and see if it works? (restart your notebook I mean)

Pip already worked for me on notebook1004, but I found out I had something in my shell config modifying the path. I've deleted the config file and restarted my server, and it still works in the Jupyter terminal.

notebook1003 already worked, but I didn't find anything in my shell config could be responsible 🤷‍♂️

Ok I think I have finally some clue about what's happening. When I open a terminal in Jupyter, this happens:

elukey   29218  0.1  0.1 1521272 75228 ?       Ssl  06:12   0:04 /home/elukey/venv/bin/python3 /home/elukey/venv/bin/jupyterhub-singleuser --port=38847
elukey   29267  0.0  0.0   7916  4796 pts/5    Ss+  06:13   0:00  \_ /bin/bash -l

As you can see, the default is to run bash -l that is meant to start a shell for the Jupyter user as if it was its own one (so as if it was just logged in). This allows any user to set up their own bashrc/local/etc.. configs and get them picked up in the Terminal, but of course it seems to prevent our dear pip to work by default.

Please note: it seems that pip works fine in the Python kernels (when doing commands like !pip etc..), it is only a problem for the Terminal.

@nshahquinn-wmf was this issue resolved?

In T247752#6086848, @Nuria wrote:

@nshahquinn-wmf was this issue resolved?

Issue is still WIP, I am trying to figure out if there is a workaround :)

FYI, @Mayakp.wiki was blocked by this when trying to switch one of the stat machines in response to T249752. Other people will probably have the same experience.

In T247752#5975529, @mpopov wrote:

I just have export PATH=/home/bearloga/venv/bin:${PATH} in my ~.bashrc file on all the machines.

For some reason, this didn't work for until I put the command in ~/.profile or ~/.bash_profile. I didn't try restarting my Jupyter server, so that could be the reason, although it doesn't seem very likely.

In T247752#6179982, @nshahquinn-wmf wrote:

For some reason, this didn't work for until I put the command in ~/.profile or ~/.bash_profile. I didn't try restarting my Jupyter server, so that could be the reason, although it doesn't seem very likely.

I would recommend also doing the following from https://meta.wikimedia.org/wiki/User:MPopov_(WMF)/Notes/Setup#Python_environment:

I also have a ~./bash_profile containing [[ -r ~/.bashrc ]] && . ~/.bashrc because sometimes bashrc does not get sourced upon connection.