Not yet clear if it is just that I made an impressively deep mess of my env or if there is an update (likely to the jupyterhub chart) that is keeping it from starting. At any rate the dev env started failing to load for me yesterday. I've so far:
removed all docker containers, images, and volumes. Reloaded the jupyterhub chart. Tried a different user in a private window. Removed all of docker itself and re-installed it. I get the following error out of the hub pod when I try to launch my server:
[I 2022-06-10 14:17:02.699 JupyterHub provider:574] Creating oauth client jupyterhub-user-VRook_%28WMF%29
[I 2022-06-10 14:17:02.706 JupyterHub log:189] 302 GET /hub/spawn/VRook_%28WMF%29 -> /hub/spawn-pending/VRook_%28WMF%29 (VRook_(WMF)@192.168.49.1) 27.00ms
[E 2022-06-10 14:17:02.707 JupyterHub user:718] Unhandled error starting VRook_(WMF)'s server: 'MYSQL_SERVICE_HOST'
[W 2022-06-10 14:17:02.718 JupyterHub base:976] 4 consecutive spawns failed. Hub will exit if failure count reaches 5 before succeeding
[E 2022-06-10 14:17:02.718 JupyterHub gen:623] Exception in Future <Task finished name='Task-78' coro=<BaseHandler.spawn_single_user.<locals>.finish_user_spawn() done, defined at /usr/local/lib/python3.8/dist-packages/jupyterhub/handlers/base.py:900> exception=KeyError('MYSQL_SERVICE_HOST')> after timeout
Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/tornado/gen.py", line 618, in error_callback
future.result()
File "/usr/local/lib/python3.8/dist-packages/jupyterhub/handlers/base.py", line 907, in finish_user_spawn
await spawn_future
File "/usr/local/lib/python3.8/dist-packages/jupyterhub/user.py", line 736, in spawn
raise e
File "/usr/local/lib/python3.8/dist-packages/jupyterhub/user.py", line 613, in spawn
await maybe_future(authenticator.pre_spawn_start(self, spawner))
File "<string>", line 90, in pre_spawn_start
File "/usr/lib/python3.8/os.py", line 675, in __getitem__
raise KeyError(key) from None
KeyError: 'MYSQL_SERVICE_HOST'
[E 2022-06-10 14:17:02.721 JupyterHub pages:380] Previous spawn for VRook_(WMF) failed: 'MYSQL_SERVICE_HOST'
[E 2022-06-10 14:17:02.722 JupyterHub log:181] {
"Cookie": "jupyterhub-hub-login=[secret]; csrf-token=[secret]; host=[secret]; user-list-id=[secret]; jupyterhub-session-id=[secret]",
"Accept-Language": "en-US,en;q=0.9",
"Accept-Encoding": "gzip, deflate",
"Referer": "http://hub.paws.local/hub/spawn-pending/VRook_%28WMF%29",
"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9",
"User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/102.0.5005.61 Safari/537.36",
"Upgrade-Insecure-Requests": "1",
"X-Scheme": "http",
"X-Forwarded-Scheme": "http",
"X-Forwarded-Proto": "http,http",
"X-Forwarded-Port": "80,80",
"X-Forwarded-Host": "hub.paws.local",
"X-Forwarded-For": "192.168.49.1,::ffff:172.17.0.2",
"X-Real-Ip": "192.168.49.1",
"X-Request-Id": "1959d2030f8f60ffa23e772bf0e404f1",
"Host": "hub.paws.local",
"Connection": "close"
}
[E 2022-06-10 14:17:02.722 JupyterHub log:189] 500 GET /hub/spawn-pending/VRook_%28WMF%29 (VRook_(WMF)@192.168.49.1) 2.93ms
[W 2022-06-10 14:17:02.744 JupyterHub log:189] 404 GET /hub/logo (@192.168.49.1) 0.59ms