There are current widespread intermittent DNS resolution issues within the Toolforge Kubernetes cluster that might have began as early as Sunday. These issues are causing some jobs and deployments to fail, particularly on NFS worker nodes.
Impact:
| Subject | Repo | Branch | Lines +/- | |
|---|---|---|---|---|
| toolforge.k8s.reboot: swap the control node if it's the one to reboot | cloud/wmcs-cookbooks | main | +11 -2 |
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Open | None | T380882 openstack network problems (November 2024) | |||
| Resolved | aborrero | T380844 2024-11-26 Toolforge DNS incident |
The current status is stable again, we are still investigating the root causes, but the cluster is up and running.
Let us know if there's any new issues arising
Change #1097991 had a related patch set uploaded (by David Caro; author: David Caro):
[cloud/wmcs-cookbooks@main] toolforge.k8s.reboot: swap the control node if it's the one to reboot
Change #1097991 merged by jenkins-bot:
[cloud/wmcs-cookbooks@main] toolforge.k8s.reboot: swap the control node if it's the one to reboot