Page MenuHomePhabricator

[bug] PAWS: User servers failing to spawn (timeout 30s)
Closed, DeclinedPublic

Description

when i try connect to jupyter notebook it gets this error (Spawn failed: Server at http://{ip:port}/user/{user}/ didn't respond in 30 seconds)

Observed logs:

[Normal] Created container notebook
[Normal] Started container notebook
Spawn failed: Server didn't respond in 30 seconds

Several users have the same issue. Spawn page hangs indefinitely and server does not start.

Requesting WMCS team to investigate the PAWS cluster / nodes.

Event Timeline

Hello there, Can you tell me that If your problem is solved or not Till now?

Waiting for response.........

Sadeiiw67 added a project: Toolforge.
Sadeiiw67 added a subscriber: fnegri.
Sadeiiw67 renamed this task from [bug] <i cant connect in my jupyter notebook> to [bug] PAWS: User servers failing to spawn (timeout 30s).Nov 23 2025, 8:35 AM
Sadeiiw67 updated the task description. (Show Details)
Sadeiiw67 raised the priority of this task from High to Unbreak Now!.Nov 23 2025, 9:23 AM
Sadeiiw67 added a project: Toolforge.
Anoop removed fnegri as the assignee of this task.Nov 23 2025, 9:40 AM
Anoop lowered the priority of this task from Unbreak Now! to Needs Triage.
Anoop removed a project: Toolforge.

@Sadeiiw67 , per https://www.mediawiki.org/wiki/Bug_management/Phabricator_etiquette you were already informed not to assign task to someone without there consent, and this task doesn't qualify for Unbreak Now as I am able to login and work on PAWS without any error, this has also been confirmed by other user in T410833#11398747

For me it's not working what can I do now? @Anoop

You don't have to do anything, please wait for someone who are can handle this request to respond

I can't currently reproduce the issue -- navigating to https://hub-paws.wmcloud.org/ spawns a container for me and works as expected. Is this still a problem @Sadeiiw67 ?

I can reproduce the issue by going to https://hub-paws.wmcloud.org/hub/spawn-pending/Haywi578 and clicking "Relaunch server"

It's failing with a "permission denied" error:

root@bastion:~# kubectl logs  -n prod jupyter--48aywi578
[I 2025-11-24 11:12:57.570 ServerApp] Extension package jupyter_server_proxy took 0.1182s to import
[I 2025-11-24 11:12:59.739 ServerApp] Extension package panel.io.jupyter_server_extension took 1.1762s to import
[I 2025-11-24 11:12:59.740 ServerApp] jupyter_lsp | extension was successfully linked.
[I 2025-11-24 11:12:59.740 ServerApp] jupyter_server_proxy | extension was successfully linked.
[I 2025-11-24 11:12:59.745 ServerApp] jupyter_server_terminals | extension was successfully linked.
[I 2025-11-24 11:12:59.746 JupyterHubSingleUser] Starting jupyterhub single-user server extension version 4.1.1
[I 2025-11-24 11:12:59.746 JupyterHubSingleUser] Using default url from server extension lab: /lab
[I 2025-11-24 11:12:59.749 ServerApp] jupyterhub | extension was successfully linked.
[W 2025-11-24 11:12:59.751 LabApp] 'extra_template_paths' was found in both NotebookApp and ServerApp. This is likely a recent change. This config will only be set in NotebookApp. Please check if you should also config these traits in ServerApp for your purpose.
[I 2025-11-24 11:12:59.754 ServerApp] jupyterlab | extension was successfully linked.
[W 2025-11-24 11:12:59.755 ServerApp] notebook_shim | error linking extension: [Errno 13] Permission denied: '/home/paws/.local/share/jupyter/runtime'

The user's home directory /srv/paws/project/paws/userhomes/79386614 was manually locked on Nov, 19 because it contains a copy of the xmrig cryptominer.

It's true I was making a tutorial for my class related to mining with xmrig and telling them how it's work and after the class I have turned off the miner because I know that it is illegal to mine here, but the folder /directory remains that why it gets locked.

Sadeiiw67 claimed this task.

As the user was trying something like mining, so its got locked, and any further action like mining will also be blocked. So please Don't try them here.

Happy Hacking! :)

Pppery changed the task status from Resolved to Declined.Nov 24 2025, 7:10 PM
Pppery removed Sadeiiw67 as the assignee of this task.

There is a real bug buried in the tide of noise here - PAWS should report the error to the user better. That is, if a user's homedir is locked it should say "You been banned from using PAWS. Contact <somewhere> if this is in error", rather than hanging indefinitely.