As a result of reboots for T383583 I caused a massive NFS outage which is requiring reboots of many toolforge workers.
Description
Description
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | Andrew | T383583 VM nova records attached to incorrect cloudcephmon IPs | |||
| Resolved | BUG REPORT | MBH | T383623 All Toolforge tool folders (and almost all user folders) disappeared | ||
| Resolved | Andrew | T383625 [2025-01-13] tools nfs outage | |||
| Resolved | dcaro | T383616 MetricsinfraAlertmanagerDown Metricsinfra alertmanager is unreachable #page |
Event Timeline
Comment Actions
Mentioned in SAL (#wikimedia-cloud-feed) [2025-01-13T20:42:57Z] <andrew@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-20 (T383625)
Comment Actions
Mentioned in SAL (#wikimedia-cloud-feed) [2025-01-13T20:43:02Z] <andrew@cloudcumin1001> END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-20 (T383625)
Comment Actions
Mentioned in SAL (#wikimedia-cloud-feed) [2025-01-13T20:43:31Z] <andrew@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-21 (T383625)
Comment Actions
Mentioned in SAL (#wikimedia-cloud-feed) [2025-01-13T20:48:54Z] <andrew@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-21 (T383625)
Comment Actions
Mentioned in SAL (#wikimedia-cloud-feed) [2025-01-13T20:52:04Z] <andrew@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-36 (T383625)
Comment Actions
Mentioned in SAL (#wikimedia-cloud-feed) [2025-01-13T20:57:28Z] <andrew@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-36 (T383625)
Comment Actions
Mentioned in SAL (#wikimedia-cloud-feed) [2025-01-13T21:05:55Z] <andrew@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-45 (T383625)
Comment Actions
Mentioned in SAL (#wikimedia-cloud-feed) [2025-01-13T21:10:40Z] <andrew@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-45 (T383625)
Comment Actions
Mentioned in SAL (#wikimedia-cloud-feed) [2025-01-13T21:14:20Z] <andrew@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-75 (T383625)
Comment Actions
Mentioned in SAL (#wikimedia-cloud-feed) [2025-01-13T21:15:21Z] <andrew@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-75 (T383625)
Comment Actions
Mentioned in SAL (#wikimedia-cloud-feed) [2025-01-13T21:18:44Z] <andrew@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-21 (T383625)
Comment Actions
Mentioned in SAL (#wikimedia-cloud-feed) [2025-01-13T21:19:32Z] <andrew@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-21 (T383625)
Comment Actions
Mentioned in SAL (#wikimedia-cloud-feed) [2025-01-13T21:20:13Z] <andrew@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-74 (T383625)
Comment Actions
Mentioned in SAL (#wikimedia-cloud-feed) [2025-01-13T21:24:54Z] <andrew@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-74 (T383625)
Comment Actions
Mentioned in SAL (#wikimedia-cloud-feed) [2025-01-13T21:25:33Z] <andrew@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-38 (T383625)
Comment Actions
Mentioned in SAL (#wikimedia-cloud-feed) [2025-01-13T21:30:13Z] <andrew@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-38 (T383625)
Comment Actions
Mentioned in SAL (#wikimedia-cloud-feed) [2025-01-13T21:31:03Z] <andrew@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-47 (T383625)
Comment Actions
Mentioned in SAL (#wikimedia-cloud-feed) [2025-01-13T21:35:45Z] <andrew@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-47 (T383625)