This usually indicates that the worker got stuck due to NFS going away, and should probably be restarted.
Currently we don't have any alerts so it's only when a user complains or we happen to check the graphs that we notice.
This alert does not need to page, but at least show up so we handle it when online.
Current dashboard: https://grafana-rw.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview?orgId=1&from=now-30m&to=now&forceLogin=true
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | BUG REPORT | Leloiandudu | T379132 chie-bot: Jobs hang on toolforge | ||
| Resolved | dcaro | T377420 [jobs-api,jobs-cli] Introduce a way to stop stuck cronjobs | |||
| Resolved | Feature | dcaro | T306391 [jobs-api] Allow Toolforge scheduled jobs to have a maximum runtime | ||
| Resolved | dcaro | T362093 [infra] Add alert when workers have a sustained large amount of D processes |
dcaro opened https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts/-/merge_requests/11
kubernetes: add workers with D stuck processes alert
dcaro merged https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts/-/merge_requests/11
kubernetes: add workers with D stuck processes alert
@dcaro just curious, what happens when the alert goes off? do you manually go and kill the worker?
Currently yes, it shows up in our alert dashboard, and whomever is oncall (or if someone sees it before that) will reboot the worker node.
Ideally we would do it automatically and just notify us, someday (:fingerscrossed:)