Page MenuHomePhabricator

Improve detection of failing ssh to toolforge bastions
Open, MediumPublic

Description

Followup from T404047: ssh login.toolforge.org was failing due to nfs on bastion being wedged, and there we no alerts raised for the issue. This task tracks improving the situation.

Possible approaches:

  1. Extend NFS checks (via D process count) from nfs-workers to bastions (i.e. add to https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts/-/blob/main/kubernetes/worker_stuck.yaml?ref_type=heads)
  2. Cover "users can't ssh into bastion" as part of toolforge-wide checks (cfr T313030, T357977 and older attempt https://gerrit.wikimedia.org/r/c/operations/puppet/+/755321/ too) via toolschecker

I (Filippo) think going for option 2 will be more future-proof since it covers actual user experience, albeit requiring more work up front.

Event Timeline

• taavi triaged this task as Medium priority.Sep 26 2025, 11:38 AM

bastion-15 was in trouble for ~1h and then something started exhausting all memory, we rebooted it after ~45m after user report of ssh failing. unclear for how long ssh logins have been failing before user report

2026-07-30-172752_2434x1285_scrot.png (2,434×1,285 px, 321 KB)

Also of note, we should also get per-systemd-unit memory usage breakdowns via cadvisor for example to know what was using all the memory