Maniphest T279480

Jupyter conda bug: cannot spawn new server
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	Ottomata
	Apr 6 2021, 9:01 PM

Tags

Referenced Files

None

Subscribers

Daniel_Mietchen

View All 19 Subscribers

Description

All of the sudden, after out and log in, jupyterhub-conda is no longer able to spawn new notebook servers. I see 2 errors:

/run is a readonly filesystem

jupyter-$USER-singleuser server is trying to write a user's environment variable file into /run, and the SystemdSpawner for it is not configured to be able to write to /run.

CondaEnvProfilesSpawner doesn't work properly.

super().options_form does not exist anymore?

https://gerrit.wikimedia.org/r/c/operations/puppet/+/677341

Should fix both of these problems, but after applying it, I get a 404 from JupyterHub when it tried to connect to the jupyter-singleuser Notebook server.

Details

	Subject	Repo	Branch	Lines +/-
	Release 2020.02~wmf5	operations/debs/anaconda-wmf	debian	+10 -5
	Release 2020.02~wmf4	operations/debs/anaconda-wmf	debian	+25 -19

Customize query in gerrit

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Resolved		Ottomata	T224658 Newpyter - SWAP Juypter Rewrite
		Resolved		Ottomata	T279480 Jupyter conda bug: cannot spawn new server

Event Timeline

Ottomata created this task.Apr 6 2021, 9:01 PM

Restricted Application removed a project: Patch-For-Review. · View Herald TranscriptApr 6 2021, 9:01 PM

Ottomata added a subscriber: • razzi.Apr 6 2021, 9:01 PM

Urbanecm subscribed.Apr 6 2021, 9:49 PM

Ah, this is why /run is being used (since the rrercent anaconda-wmf release)

https://github.com/jupyterhub/systemdspawner/commit/1b83c2738f4a3fa34b5242da9cb098a04901e6dc

FYI, I am working on this now and hope to get it fixed within a few hours. "hope" :)

Change 677645 had a related patch set uploaded (by Ottomata; author: Ottomata):

[operations/debs/anaconda-wmf@debian] Release 2020.02~wmf4

https://gerrit.wikimedia.org/r/677645

gerritbot added a project: Patch-For-Review.Apr 7 2021, 7:22 PM

Change 677645 abandoned by Ottomata:

[operations/debs/anaconda-wmf@debian] Release 2020.02~wmf4

Reason:

https://gerrit.wikimedia.org/r/677645

Change 677645 restored by Ottomata:

[operations/debs/anaconda-wmf@debian] Release 2020.02~wmf4

https://gerrit.wikimedia.org/r/677645

Change 677645 merged by Ottomata:

[operations/debs/anaconda-wmf@debian] Release 2020.02~wmf4

https://gerrit.wikimedia.org/r/677645

Mentioned in SAL (#wikimedia-analytics) [2021-04-07T19:53:00Z] <ottomata> upgrade anaconda-wmf everywhere to 2020.02~wmf4 with fixes for T279480

Unreverted and merged the puppet fix: https://gerrit.wikimedia.org/r/c/operations/puppet/+/677403

Getting the stat boxes back in place. stat1008 and stat1004 are good to go.

Along the way, I introduced another small bug for users that are creating new conda envs for the first time. Fixing that now...

Change 677669 had a related patch set uploaded (by Ottomata; author: Ottomata):

[operations/debs/anaconda-wmf@debian] Release 2020.02~wmf5

https://gerrit.wikimedia.org/r/677669

Mentioned in SAL (#wikimedia-analytics) [2021-04-07T22:51:17Z] <ottomata> installing anaconda-wmf-2020.02~wmf5 on stat boxes - T279480

Mentioned in SAL (#wikimedia-analytics) [2021-04-07T23:03:45Z] <ottomata> installing anaconda-wmf-2020.02~wmf5 on remaining nodes - T279480

Change 677669 merged by Ottomata:

[operations/debs/anaconda-wmf@debian] Release 2020.02~wmf5

https://gerrit.wikimedia.org/r/677669

Ok, should be fixed.

I'm still installing the new anaconda-wmf on all the worker nodes, etc, but the stat boxes should be good to go.

Ok, all should be back to normal!

Ottomata triaged this task as High priority.Apr 8 2021, 1:15 PM

Ottomata moved this task from Next Up to Done on the Analytics-Kanban board.

• fdans closed this task as Resolved.Apr 8 2021, 4:24 PM