Page MenuHomePhabricator

pip not accessible in new SWAP virtual environments
Open, MediumPublic

Description

I just logged into stat1004 to check out its new JupyterHub installation. Mostly, everything worked fine, but when I tried running pip (or pip3) I got an error: bash: pip: command not found.

I can still use pip by providing the full path, ~/venv/bin/pip, but that shouldn't be necessary.

Related Objects

Event Timeline

I remember @SNowick_WMF also experienced this when she first started using SWAP, and she may still be using the ~/venv/bin/pip workaround. Shay, what server have you been using?

Ottomata added a subscriber: Ottomata.EditedMar 16 2020, 3:23 PM

I just logged into stat1004 to check out its new JupyterHub installation.

Did you do this via JupyterHub in the browser, or just in the terminal via ssh? If the latter, you need to activate ~/venv like you would any normal virtualenv (source ~/venv/bin/activate); it is only automatically activated for you in your Jupyter notebook server.

nshahquinn-wmf added a comment.EditedMar 16 2020, 3:34 PM

Did you do this via JupyterHub in the browser, or just in the terminal via ssh? If the latter, you need to activate ~/venv like you would any normal virtualenv (source ~/venv/bin/activate); it is only automatically activated for you in your Jupyter notebook server.

This was using JupyterHub's in-browser terminal, so it should have worked automatically. I helped Shay troubleshoot back when I encountered this, and I know she was in the JupyterHub terminal as well.

elukey added a subscriber: elukey.EditedMar 16 2020, 3:45 PM

@nshahquinn-wmf if you have a Jupyter terminal handy, what does the following command say which pip3? Where it works as your expect I mean (I guess on notebook1003/4)

@nshahquinn-wmf if you have a Jupyter terminal handy, what does the following command say which pip3? Where it works as your expect I mean (I guess on notebook1003/4)

Sure!

neilpquinn-wmf@notebook1003:~$ which pip3
/home/neilpquinn-wmf/venv/bin/pip3

I remember @SNowick_WMF also experienced this when she first started using SWAP, and she may still be using the ~/venv/bin/pip workaround. Shay, what server have you been using?

@SNowick_WMF encountered this on notebook1004, so it seems to have to do with new Jupyter environments rather than specific servers.

mpopov added a subscriber: mpopov.Mar 17 2020, 2:17 PM

I just have export PATH=/home/bearloga/venv/bin:${PATH} in my ~.bashrc file on all the machines.

@elukey @Ottomata: perhaps that sort of thing can be done automatically for users as part of the initial homedir setup??

I use notebook1004, to fix this I added aliases in my .bash_profile and
haven't had any issues since.

alias py='/home/snowick/venv/bin/python3'
alias pip='/home/snowick/venv/bin/pip'
alias jupyter='/home/snowick/venv/bin/jupyter'

I think that Neil is right, it is probably something that I missed when upgrading SystemdSpawner on all the notebooks/stats. For example, on notebook1003:

elukey@notebook1003:~$ echo $PATH
/home/elukey/venv/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

At the beginning of the PATH there is the venv's bin, that should come from this directive in Jupyterhub:

c.SystemdSpawner.extra_paths = ['/home/{USERNAME}/venv/bin']

On notebook1003 works, on the rest of the node it doesn't. This is probably a PEBCAK from my side, will try to find what I missed (apologies).

Ok so the $PATH is correctly picked up in kernels (I tested a py3 one) but not from the Terminal. Not sure what different on notebook1003, it has been upgraded as well like 1004.

For some mysterious reasons, I can now see PATH properly populated in notebook1004:

elukey@notebook1004:~$ echo $PATH
/home/elukey/venv/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

I have restarted jupyterhub and also my own notebook. @nshahquinn-wmf would you mind to try on 1004 and see if it works? (restart your notebook I mean)

Ok so the $PATH is correctly picked up in kernels (I tested a py3 one) but not from the Terminal. Not sure what different on notebook1003, it has been upgraded as well like 1004.

Ah, yeah, I just tried in a Python notebook on stat1004, and I was able to use pip without problems.

@nshahquinn-wmf would you mind to try on 1004 and see if it works? (restart your notebook I mean)

Pip already worked for me on notebook1004, but I found out I had something in my shell config modifying the path. I've deleted the config file and restarted my server, and it still works in the Jupyter terminal.

notebook1003 already worked, but I didn't find anything in my shell config could be responsible 🤷‍♂️

Ottomata assigned this task to elukey.Mar 23 2020, 3:40 PM
Ottomata added a project: Analytics-Kanban.
Ottomata moved this task from Next Up to Done on the Analytics-Kanban board.
Ottomata moved this task from Incoming to Operational Excellence on the Analytics board.
Nuria closed this task as Resolved.Apr 9 2020, 5:01 PM
nshahquinn-wmf reopened this task as Open.Apr 10 2020, 4:35 PM
nshahquinn-wmf moved this task from Operational Excellence to Incoming on the Analytics board.

Actually, this is still happening. I just tested in the Jupyter terminals on stat1004 and stat1005, and encountered the same error.

Milimetric triaged this task as High priority.Apr 13 2020, 3:28 PM
Milimetric moved this task from Incoming to Operational Excellence on the Analytics board.

Ok I think I have finally some clue about what's happening. When I open a terminal in Jupyter, this happens:

elukey   29218  0.1  0.1 1521272 75228 ?       Ssl  06:12   0:04 /home/elukey/venv/bin/python3 /home/elukey/venv/bin/jupyterhub-singleuser --port=38847
elukey   29267  0.0  0.0   7916  4796 pts/5    Ss+  06:13   0:00  \_ /bin/bash -l

As you can see, the default is to run bash -l that is meant to start a shell for the Jupyter user as if it was its own one (so as if it was just logged in). This allows any user to set up their own bashrc/local/etc.. configs and get them picked up in the Terminal, but of course it seems to prevent our dear pip to work by default.

Please note: it seems that pip works fine in the Python kernels (when doing commands like !pip etc..), it is only a problem for the Terminal.

Nuria added a subscriber: Nuria.Apr 27 2020, 8:31 PM

@nshahquinn-wmf was this issue resolved?

@nshahquinn-wmf was this issue resolved?

Issue is still WIP, I am trying to figure out if there is a workaround :)

Nuria moved this task from Done to In Progress on the Analytics-Kanban board.May 4 2020, 7:19 PM

FYI, @Mayakp.wiki was blocked by this when trying to switch one of the stat machines in response to T249752. Other people will probably have the same experience.

I just have export PATH=/home/bearloga/venv/bin:${PATH} in my ~.bashrc file on all the machines.

For some reason, this didn't work for until I put the command in ~/.profile or ~/.bash_profile. I didn't try restarting my Jupyter server, so that could be the reason, although it doesn't seem very likely.

mpopov added a comment.Jun 1 2020, 2:47 PM

For some reason, this didn't work for until I put the command in ~/.profile or ~/.bash_profile. I didn't try restarting my Jupyter server, so that could be the reason, although it doesn't seem very likely.

I would recommend also doing the following from https://meta.wikimedia.org/wiki/User:MPopov_(WMF)/Notes/Setup#Python_environment:

I also have a ~./bash_profile containing [[ -r ~/.bashrc ]] && . ~/.bashrc because sometimes bashrc does not get sourced upon connection.

Milimetric lowered the priority of this task from High to Medium.Jun 8 2020, 4:45 PM
elukey moved this task from Backlog to Q1 2020/2021 on the Analytics-Clusters board.
Aklapper removed a project: Analytics.Sat, Jul 4, 7:59 AM