This usually indicates that the worker got stuck due to NFS going away, and should probably be restarted.
Currently we don't have any alerts so it's only when a user complains or we happen to check the graphs that we notice.
This alert does not need to page, but at least show up so we handle it when online.
Current dashboard: https://grafana-rw.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview?orgId=1&from=now-30m&to=now&forceLogin=true
| Title | Reference | Author | Source Branch | Dest Branch | |
|---|---|---|---|---|---|
| kubernetes: add workers with D stuck processes alert | repos/cloud/toolforge/alerts!11 | dcaro | add_d_processes_alert | main |
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Stalled | Feature | None | T306391 [jobs-api] Allow Toolforge scheduled jobs to have a maximum runtime | ||
| Resolved | dcaro | T362093 [infra] Add alert when workers have a sustained large amount of D processes |
dcaro opened https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts/-/merge_requests/11
kubernetes: add workers with D stuck processes alert
dcaro merged https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts/-/merge_requests/11
kubernetes: add workers with D stuck processes alert