Page MenuHomePhabricator

PAWS fails to spawn user container
Closed, ResolvedPublic


Paws is failing to spawn a user's container. Hub log shows a line after timeout:

JupyterHub user:421] USERNAME server failed to start in 300 seconds, giving up

Judging by the hub logs it seems 12 users are affected by this right now.

Event Timeline

Chicocvenancio created this task.
Chicocvenancio lowered the priority of this task from High to Low.Feb 22 2018, 5:56 PM

Seems to be a result of many idle user pods staying active as a result of T175202.

I manually started the culler and got things working again (at least one previously affected user has gotten a pod started):

(using literal block to preserve backtick in command)

yuvipanda@tools-paws-master-01:~$ kubectl --namespace=prod exec -it hub-649d9bb8b4-92tl7 /bin/bash
root@hub-649d9bb8b4-92tl7: export JUPYTERHUB_API_TOKEN= %`jupterhub token yuvipanda`
root@hub-649d9bb8b4-92tl7: /usr/local/bin/ --timeout=3600 --cull_every=600 --url=

@Barcex , @Zoranzoki21 could you confirm PAWS is working for you now? I'll leave this open until you do.

zhuyifei1999 closed this task as Resolved.EditedFeb 22 2018, 10:11 PM

Root cause T175202 was fixed by @Chicocvenancio , and released the resources necessary to start new pods.