Page MenuHomePhabricator

Jupyter conda bug: cannot spawn new server
Closed, ResolvedPublic


All of the sudden, after out and log in, jupyterhub-conda is no longer able to spawn new notebook servers. I see 2 errors:

  1. /run is a readonly filesystem

jupyter-$USER-singleuser server is trying to write a user's environment variable file into /run, and the SystemdSpawner for it is not configured to be able to write to /run.

  1. CondaEnvProfilesSpawner doesn't work properly.

super().options_form does not exist anymore?

Should fix both of these problems, but after applying it, I get a 404 from JupyterHub when it tried to connect to the jupyter-singleuser Notebook server.

Related Objects

Event Timeline

FYI, I am working on this now and hope to get it fixed within a few hours. "hope" :)

Change 677645 had a related patch set uploaded (by Ottomata; author: Ottomata):

[operations/debs/anaconda-wmf@debian] Release 2020.02~wmf4

Change 677645 abandoned by Ottomata:

[operations/debs/anaconda-wmf@debian] Release 2020.02~wmf4


Change 677645 restored by Ottomata:

[operations/debs/anaconda-wmf@debian] Release 2020.02~wmf4

Change 677645 merged by Ottomata:

[operations/debs/anaconda-wmf@debian] Release 2020.02~wmf4

Mentioned in SAL (#wikimedia-analytics) [2021-04-07T19:53:00Z] <ottomata> upgrade anaconda-wmf everywhere to 2020.02~wmf4 with fixes for T279480

Getting the stat boxes back in place. stat1008 and stat1004 are good to go.

Along the way, I introduced another small bug for users that are creating new conda envs for the first time. Fixing that now...

Change 677669 had a related patch set uploaded (by Ottomata; author: Ottomata):

[operations/debs/anaconda-wmf@debian] Release 2020.02~wmf5

Mentioned in SAL (#wikimedia-analytics) [2021-04-07T22:51:17Z] <ottomata> installing anaconda-wmf-2020.02~wmf5 on stat boxes - T279480

Mentioned in SAL (#wikimedia-analytics) [2021-04-07T23:03:45Z] <ottomata> installing anaconda-wmf-2020.02~wmf5 on remaining nodes - T279480

Change 677669 merged by Ottomata:

[operations/debs/anaconda-wmf@debian] Release 2020.02~wmf5

Ok, should be fixed.

I'm still installing the new anaconda-wmf on all the worker nodes, etc, but the stat boxes should be good to go.

Ok, all should be back to normal!

Ottomata triaged this task as High priority.Apr 8 2021, 1:15 PM
Ottomata moved this task from Next Up to Done on the Analytics-Kanban board.