From alertmanager (https://prometheus-alerts.wmcloud.org):
alertname: ToolsGridQueueProblem summary: Grid queue webgrid-lighttpd@tools-sgeweblight-10-20.tools.eqiad1.wikimedia.cloud is in state E 9 hours ago host: tools-sgeweblight-10-20.tools.eqiad1.wikimedia.cloud instance: tools-sgegrid-master job: node queue: webgrid-lighttpd severity: warn state: E @receiver: cloud-admin-feed runbook
Mentioned in SAL (#wikimedia-cloud-feed) [2022-11-01T09:37:27Z] <wm-bot2> cleaned up grid queue errors on tools-sgegrid-master (T322110) - cookbook ran by dcaro@vulcanus
Checked the status with:
dcaro@vulcanus$ cookbook wmcs.toolforge.grid.get_cluster_status --only-failed --project tools
(and using https://gerrit.wikimedia.org/r/c/operations/cookbooks/+/841930)
that showed that there was only one error, and it was due to the epilog issue, so just cleaned up the queues and everything is back to normal.