Paws is failing to spawn a user's container. Hub log shows a line after timeout:
JupyterHub user:421] USERNAME server failed to start in 300 seconds, giving up
Judging by the hub logs it seems 12 users are affected by this right now.
Description
Description
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | Chicocvenancio | T185434 PAWS fails creating a server for new user | |||
| Resolved | Chicocvenancio | T188018 PAWS fails to spawn user container |
Event Timeline
Comment Actions
Seems to be a result of many idle user pods staying active as a result of T175202.
I manually started the culler and got things working again (at least one previously affected user has gotten a pod started):
(using literal block to preserve backtick in command)
yuvipanda@tools-paws-master-01:~$ kubectl --namespace=prod exec -it hub-649d9bb8b4-92tl7 /bin/bash
root@hub-649d9bb8b4-92tl7: export JUPYTERHUB_API_TOKEN= %`jupterhub token yuvipanda`
root@hub-649d9bb8b4-92tl7: /usr/local/bin/cull_idle_servers.py --timeout=3600 --cull_every=600 --url=http://127.0.0.1:8081/paws/hub/api
@Barcex , @Zoranzoki21 could you confirm PAWS is working for you now? I'll leave this open until you do.
Comment Actions
Root cause T175202 was fixed by @Chicocvenancio , and released the resources necessary to start new pods.