toolforge ssh login hangs right before prompt
Closed, ResolvedPublicBUG REPORT
Actions

Assigned To

Authored By

	Magnus
	Sep 9 2025, 8:50 AM

Description

Steps to replicate the issue (include links if applicable):

What happens?:
Login successful, login screen shows, but hangs right before showing the prompt:

...
Last Puppet commit: (aa7682114c) Muehlenhoff - Update DHCP server in esams
Last login: Tue Sep  9 08:08:09 2025 from 51.191.195.147

Ctrl-C does not do anything.

What should have happened instead?:
Prompt.

FWIW I can ssh into bastion.wmflabs.org but when I try to ssh into login.toolforge.org from there, same thing.

Mentioned In: T404584: [tools,nfs,infra] Address tools NFS getting stuck with processes in D state
T404054: Improve detection of failing ssh to toolforge bastions
Mentioned Here: T404054: Improve detection of failing ssh to toolforge bastions

Restricted Application added a subscriber: Aklapper. · View Herald Transcript

Looking

NFS went away, rebooting:

[Tue Sep  9 05:02:50 2025] nfs: server tools-nfs.svc.tools.eqiad1.wikimedia.cloud not responding, still trying

Mentioned in SAL (#wikimedia-cloud-feed) [2025-09-09T08:55:10Z] <wmbot~dcaro@acme> START - Cookbook wmcs.vps.instance.force_reboot (T404047)

Mentioned in SAL (#wikimedia-cloud-feed) [2025-09-09T08:55:17Z] <wmbot~dcaro@acme> END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) (T404047)