backup2014: should be ok without depooling for a short period
ms-backup2004 will require manual stopping of backup processes there, I will take care of that before it happens.

Mentioned in SAL (#wikimedia-operations) [2026-07-06T11:07:23Z] <moritzm> failover Ganeti master in codfw to ganeti2032 T430909

Draining ganeti2048.codfw.wmnet of running VMs

Thanks for your patience while I was out last week, @ayounsi.

In short, conf2004 (or any conf* host) will require manual coordination between Service Ops and (to some extent) Traffic.

Points of note:

• conf2004 is the statically configured codfw PyBal etcd host.
• Although we tend to allow "transient" connectivity disruptions (measured in seconds, like a ToR switch migration) to happen "hot" for certain classes of clients (i.e., clients other than PyBal or etcd-mirror, such as confd and MediaWiki), 20m is a rather long time for a cluster member to be isolated. For example, I have concerns about the brief period where it becomes reachable again to clients, but has not yet caught up from its peers - particularly for clients that do not use quorum reads (e.g., MediaWiki).
• There's also the collocated Zookeeper cluster to consider (used by, e.g., Kafka).

I think the cleanest option is to depool conftool etcd clients from codfw, directing them to eqiad for the duration of the maintenance, while there's not really much we can do for Zookeeper.

Procedure

For etcd, this is quite similar to https://wikitech.wikimedia.org/wiki/Etcd/Main_cluster#Reimage_cluster (i.e., what we do in advance of starting the reimages).

For Zookeeper, there is no equivalent operation - i.e., there is nowhere to depool to. What I'd propose here is simply confirming whether conf2004 happens to be the Zookeeper leader at the time (e.g., hit 2181 with stat) and if so, consider gracefully moving leadership away preemptively (the only way I know how to do that is via a service restart, but I'm open to suggestions). Having a single Zookeeper ensemble member unavailable for a time should be okay, but we probably still want to keep an eye out for Kafka-related sadness in codfw.

Depool

1. PyBals (coordinate with Traffic)
   • Update profile::pybal::config_host in hieradata/role/codfw/lvs/balancer.yaml to point to an eqiad conf* host.
   • Run puppet on the codfw LVS hosts and apply with the sre.loadbalancer.restart-pybal cookbook.
2. Other clients
   • Switch the RO-client (_etcd._tcp and _etcd-client-ssl._tcp) SRV records for codfw-associated DCs (codfw, ulsfo, eqsin) to point to their eqiad counterparts in operations/dns.
   • Wait until at least 5m passes (or manually flush recursor caches).
   • Restart confds in those same DCs (via cumin, e.g., with 'P{C:confd} and (A:ulsfo or A:eqsin or A:codfw)').
   • Restart navtiming on webperf2003.
   • Restart Liberica daemons in ulsfo and eqsin (e.g., cookbook sre.loadbalancer.upgrade -t T430909 --seamless --alias liberica-ulsfo --reason 'Clear control-plane connections to etcd' restart)
3. Verify traffic is gone, with the exception of conf2005-local traffic from etcd-mirror (note: this will not appear in nginx access logs).

Repool

After verifying conf2004 has caught up and is working as expected (e.g., no lingering errors in etcd.service logs on either 2004 or peers, /health and /v2/members on 2004 look right), perform the same procedure in reverse - i.e., reverting the earlier changes to puppet and operations/dns, and applying said reverts the same way.

Additional thoughts

So, although this is the "easy" version of the procedure because codfw is the read-only DC, it's still rather involved. We would most likely need to complete the depool over the course of the afternoon (Americas) on Tuesday (tomorrow) in order to clear the way for this to happen early UTC Wednesday.

@ayounsi - Just to confirm, how "fixed" is the tentative schedule? i.e., if we can't get everything in order that quickly, or guarantee Service Ops hands available during the maintenance itself, how disruptive would it be to defer?

Adding @JMeybohm as an additional point of contact in a time zone that overlaps with the planned work (though he may not be available on Wednesday specifically).

@ayounsi - Just to confirm, how "fixed" is the tentative schedule? i.e., if we can't get everything in order that quickly, or guarantee Service Ops hands available during the maintenance itself, how disruptive would it be to defer?

I can push it a bit later in the day (EU time) if it's better, and we can of course cancel/reschedule it if there is a hard blocker. But with the current schedule and some services already depooled it would not be ideal.

Change #1308114 had a related patch set uploaded (by Scott French; author: Scott French):

[operations/dns@master] wmnet: Temporarily direct codfw, eqsin, ulsfo etcd clients to eqiad

https://gerrit.wikimedia.org/r/1308114

Change #1308115 had a related patch set uploaded (by Scott French; author: Scott French):

[operations/puppet@production] hieradata: Temporarily point codfw PyBals at eqiad etcd

https://gerrit.wikimedia.org/r/1308115

Thanks @ayounsi - I'll aim to have this complete during my day today and will follow up here once it's safe to proceed. While no action should be required from Service Ops immediately prior to the maintenance, if you have any last-minute questions for us, @Clement_Goubert should likely be reachable around that time.

Assuming all goes well, I'll plan to execute the repool during my (UTC-7) morning on Wednesday.

Change #1308115 merged by Ssingh:

[operations/puppet@production] hieradata: Temporarily point codfw PyBals at eqiad etcd

https://gerrit.wikimedia.org/r/1308115

Change #1308114 merged by Scott French:

[operations/dns@master] wmnet: Temporarily direct codfw, eqsin, ulsfo etcd clients to eqiad

https://gerrit.wikimedia.org/r/1308114

Mentioned in SAL (#wikimedia-operations) [2026-07-07T17:40:48Z] <sukhe> restart pybal on lvs2014 to switch from conf2004 to conf1008: T430909

Mentioned in SAL (#wikimedia-operations) [2026-07-07T17:44:05Z] <swfrench-wmf> switched codfw, eqsin, ulsfo etcd client SRV records to eqiad - T430909

Mentioned in SAL (#wikimedia-operations) [2026-07-07T17:46:37Z] <sukhe> restart pybal on lvs2013 to switch from conf2004 to conf1008: T430909

Mentioned in SAL (#wikimedia-operations) [2026-07-07T17:51:00Z] <sukhe> restart pybal on lvs2012 to switch from conf2004 to conf1008 [puppet re-enabled there]: T430909

Mentioned in SAL (#wikimedia-operations) [2026-07-07T17:52:40Z] <sukhe> restart pybal on lvs2011 to switch from conf2004 to conf1008: T430909

Mentioned in SAL (#wikimedia-operations) [2026-07-07T17:59:14Z] <swfrench-wmf> restarted ulsfo confds, confirmed now connected to eqiad backends - T430909

Mentioned in SAL (#wikimedia-operations) [2026-07-07T18:01:13Z] <swfrench-wmf> restarted navtiming on webperf2003 - T430909

Mentioned in SAL (#wikimedia-operations) [2026-07-07T18:11:53Z] <swfrench-wmf> restarted eqsin, codfw confds - T430909

Mentioned in SAL (#wikimedia-operations) [2026-07-07T18:38:50Z] <swfrench@cumin2002> START - Cookbook sre.loadbalancer.upgrade restart A:liberica-ulsfo (T430909)

Mentioned in SAL (#wikimedia-operations) [2026-07-07T18:40:41Z] <swfrench@cumin2002> END (PASS) - Cookbook sre.loadbalancer.upgrade (exit_code=0) restart A:liberica-ulsfo (T430909)

Mentioned in SAL (#wikimedia-operations) [2026-07-07T18:49:09Z] <swfrench@cumin2002> START - Cookbook sre.loadbalancer.upgrade restart A:liberica-eqsin (T430909)

Mentioned in SAL (#wikimedia-operations) [2026-07-07T18:52:08Z] <swfrench@cumin2002> END (PASS) - Cookbook sre.loadbalancer.upgrade (exit_code=0) restart A:liberica-eqsin (T430909)

Status

Alright, the (updated) procedure in T430909#12091874 is complete, and since ~ 18:51 UTC the codfw etcd cluster is receiving no traffic. All that remains is /metrics scraping from prometheus* hosts and etcd-mirror traffic internal to conf2005 (those are the tiny tail of PUT/DELETE heavy ops visible in the etcd dashboard).

No obvious issues with the eqiad cluster running under increased read load, nor reported by MediaWiki now that etcd config fetch is happening over the WAN (i.e., with a 30ms+ RTT). The only visible effect is an increase in PHP Notice: EtcdConfig using stale data: lost lock log spam in codfw in the PHP-FPM errorlog. This makes sense, since the latency of the fetch-enclosing critical section is now higher, so when the cached config becomes eligible for refresh, we'll see a higher rate of workers attempting to fetch, only to discover another worker is already doing so.

Next steps

I believe we're good to go for the rack maintenance without further action, other than downtiming the host, which is presumably handled as part of the rack maintenance process.

I've double checked just now that conf2004 is not the current Zookeeper leader in the codfw cluster (conf2005 is). Given that leadership tends to stay fairly stable, let's assume we don't need to take additional action there (i.e., to preemptively force it away).

As noted before, the only area where I'm a bit hazy on what to expect while conf2004 is isolated is how Kafka's Zookeeper client responds. Given that it seems we don't do anything special for Zookeeper during conf* reimages, this probably isn't something we need to take action on (but is worth being aware of in the event something does happen).

Assuming all goes according to plan, I'll follow up during my day tomorrow to repool the cluster.

Edit: Alright, there is now one additional issue to call out: It seems conf2004 has now become the leader of the codfw etcd cluster.

As soon as it's isolated and peers start observing missed heartbeats (or, more specifically, once that happens and the ~ 1s election timeout passes), we should see a new election trigger and leadership move to one of the peers. In other words, raft does raft things, and everything should be fine in the end.

However, there's one edge case: We're (intentionally) still replicating from eqiad to codfw via etcd-mirror (happening on conf2005). If we get extremely unlucky and etcd-mirror tries to apply an op concurrently with that timeout / leader flip, the op may fail and etcd-mirror terminate, triggering the EtcdReplicationDown paging alert.

If that happens, just try starting it again (i.e., run systemctl start etcdmirror--eqiad-wmnet.service on conf2005) as described in https://wikitech.wikimedia.org/wiki/Etcd/Main_cluster#Recovering_from_replication_failures. Alternatively, if you run into other trouble, just silence the alert and I'll get things back in order when I'm online (this is fine, since no clients are using the cluster; we just need to make sure it's fixed before I repool it).

One alternative is to preemptively move leadership off of conf2004 to one of its peers, with something like (run locally from one of the conf* nodes):

ETCDCTL_API=3 etcdctl --endpoints https://$(hostname -f):2379 move-leader bc455dd2ab948ee  # bc455dd2ab948ee is conf2006

I don't think that's worth the hassle, though.

Another semi-alternative (in that, we'll still need to deal with it) is to preemptively silence EtcdReplicationDown. Not sure that's worth it either, as it just kicks the "fix it" can down the road.

In T430909#12096739, @Scott_French wrote:

Status

Alright, the (updated) procedure in T430909#12091874 is complete, and since ~ 18:51 UTC the codfw etcd cluster is receiving no traffic. All that remains is /metrics scraping from prometheus* hosts and etcd-mirror traffic internal to conf2005 (those are the tiny tail of PUT/DELETE heavy ops visible in the etcd dashboard).

[...]

Assuming all goes according to plan, I'll follow up during my day tomorrow to repool the cluster.

Thanks for the excellent work and the detailed write-up here as usual, @Scott_French!

Icinga downtime and Alertmanager silence (ID=5e02ef3f-9732-4694-88a1-3e0f414cfe47) set by ayounsi@cumin1003 for 2:00:00 on 3 host(s) and their services with reason: Switch maintenance

lsw1-b3-codfw,lsw1-b3-codfw IPv6,lsw1-b3-codfw.mgmt

Mentioned in SAL (#wikimedia-operations) [2026-07-08T08:19:19Z] <XioNoX> lsw1-b3-codfw> request system reboot - T430909

Change #1308651 had a related patch set uploaded (by Scott French; author: Scott French):

[operations/dns@master] wmnet: Revert codfw, eqsin, ulsfo etcd clients to codfw

https://gerrit.wikimedia.org/r/1308651

Change #1308653 had a related patch set uploaded (by Scott French; author: Scott French):

[operations/puppet@production] hieradata: Revert codfw PyBals at codfw etcd

https://gerrit.wikimedia.org/r/1308653

Change #1308651 merged by Scott French:

[operations/dns@master] wmnet: Revert codfw, eqsin, ulsfo etcd clients to codfw

https://gerrit.wikimedia.org/r/1308651

Change #1308653 merged by Ssingh:

[operations/puppet@production] hieradata: Revert codfw PyBals at codfw etcd

https://gerrit.wikimedia.org/r/1308653

Mentioned in SAL (#wikimedia-operations) [2026-07-08T14:34:14Z] <swfrench-wmf> switched codfw, eqsin, ulsfo etcd client SRV records back to codfw - T430909

Mentioned in SAL (#wikimedia-operations) [2026-07-08T14:52:28Z] <swfrench-wmf> restarted ulsfo confds, confirmed now connected to codfw backends except those using wikimedia.org SRV record - T430909

Mentioned in SAL (#wikimedia-operations) [2026-07-08T15:03:04Z] <swfrench-wmf> restarted eqsin, codfw confds - T430909

Mentioned in SAL (#wikimedia-operations) [2026-07-08T15:03:15Z] <sukhe@cumin1003> START - Cookbook sre.loadbalancer.upgrade restart A:liberica-ulsfo (T430909)

Mentioned in SAL (#wikimedia-operations) [2026-07-08T15:03:31Z] <swfrench-wmf> restarted navtiming on webperf2003 - T430909

Mentioned in SAL (#wikimedia-operations) [2026-07-08T15:05:04Z] <sukhe@cumin1003> END (PASS) - Cookbook sre.loadbalancer.upgrade (exit_code=0) restart A:liberica-ulsfo (T430909)

Mentioned in SAL (#wikimedia-operations) [2026-07-08T15:06:27Z] <sukhe@cumin1003> START - Cookbook sre.loadbalancer.upgrade restart A:liberica-eqsin (T430909)

Mentioned in SAL (#wikimedia-operations) [2026-07-08T15:10:41Z] <sukhe@cumin1003> END (PASS) - Cookbook sre.loadbalancer.upgrade (exit_code=0) restart A:liberica-eqsin (T430909)

The codfw etcd cluster is serving traffic as usual again. No issues encountered. I'll be refreshing our docs to reflect the procedure used here (T430909#12091874) along with some incidental discoveries along the way (e.g., the existence of the _etcd-client-ssl._tcp.wikimedia.org SRV record). Many thanks to @ssingh for the assistance with PyBal and Liberica!

Docs refresh: https://wikitech.wikimedia.org/wiki/Etcd/Main_cluster#Depool_etcd_client_traffic_from_a_cluster