I noticed that they'd both ended up on the same hypervisor today (cloudvirt1021). I moved tools-k8s-haproxy02 to cloudvirt1017 to correct the issue, but letting the scheduler figure that out by placing them both in an anti-affinity group would be better than adding them to the spread alert, I think, (though we could do that too).
Description
Description
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Open | None | T262350 bad failure cases for wmcs custom puppet enc | |||
| Resolved | taavi | T267082 Rebuild Toolforge servers that should not have NFS mounted (and with affinity) | |||
| Resolved | taavi | T252239 Rebuild tools-k8s-haproxy-* as an anti-affinity server group |
Event Timeline
Comment Actions
Mentioned in SAL (#wikimedia-cloud) [2021-05-10T14:57:38Z] <arturo> allow tools-k8s-haproxy-[3-4] to use the tools-k8s-haproxy-keepalived-vip address (172.16.6.113) (T252239)
Comment Actions
Mentioned in SAL (#wikimedia-cloud) [2021-05-10T15:22:43Z] <Majavah> change k8s.svc.tools.eqiad1.wikimedia.cloud. to point to the tools-k8s-haproxy-keepalived-vip address 172.16.6.113 (T252239)
Comment Actions
Mentioned in SAL (#wikimedia-cloud) [2021-05-11T16:29:46Z] <Majavah> carefully shutdown tools-k8s-haproxy-2 T252239
Comment Actions
Mentioned in SAL (#wikimedia-cloud) [2021-05-11T16:32:39Z] <Majavah> carefully shutdown tools-k8s-haproxy-1 T252239