Do you mean in PAWS (in Cloud VPS), or SWAP, in Analytics Production?

I suspect you mean SWAP, since I just did T230724: Upgrade all SWAP users to JupyterLab 1.0 this morning.

Hello, yes, I mean SWAP. I will update this ticket to remove PAWS. Thank you.

Something is definitely not working for me on notebook1003, am investigating. In the meantime, can you check if JupyterLab works for you on notebook1004?

no luck on notebook4, unfortunately

I rolled back to JupyterLab 0.32.1.

Icinga told us that the systemd state on both notebook1003 and notebook1004 was degraded.

Looking at the service that failed i saw:

● jupyter-iflorez-singleuser.service  loaded failed failed /bin/bash -c cd /home/iflorez && exec jupyterhub-singleuser --po
● jupyter-isaacj-singleuser.service   loaded failed failed /bin/bash -c cd /home/isaacj && exec jupyterhub-singleuser --por
● jupyter-nuria-singleuser.service    loaded failed failed /bin/bash -c cd /home/nuria && exec jupyterhub-singleuser --port
● jupyter-piccardi-singleuser.service loaded failed failed /bin/bash -c cd /home/piccardi && exec jupyterhub-singleuser --p
● jupyter-ryanmax-singleuser.service  loaded failed failed /bin/bash -c cd /home/ryanmax && exec jupyterhub-singleuser --po

I started the one for @Iflorez and it is running now. Did that fix it on notebook1003?

Mentioned in SAL (#wikimedia-operations) [2019-08-28T09:28:00Z] <mutante> notebook1004 - systemctl start jupyter-ebernhardson-singleuser (T231365)

on notebook1004 there was only 1 failed service, the one for @EBernhardson which i started. That made Icinga alerts recover.

On notebook1003 though starting the remaining failed services does not work. they keep failing.

https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=notebook1003&service=Check+systemd+state

Sorry, was talking with Iflorez in IRC last night and didn't update here. My upgrade (and downgrade) clearly had some problems. I'm working on this.

I think the issue had to do with the pip install command I was using to upgrade/rollback user venvs. I was using --ignore-installed instead of --force-reinstall. --ignore-installed will not remove old versions, just overwrite them. I think this was leaving old/bad package files in place.