@MoritzMuehlenhoff - How challenging would be to rebuild the forward-port of zookeeper (3.4.13) from bullseye that you prepared in T418915#11851872 for bookworm as well? That would be the last moving part of this intermediate in-place step to bookworm before stepping to trixie.

In T428495#12007498, @Scott_French wrote:

@MoritzMuehlenhoff - How challenging would be to rebuild the forward-port of zookeeper (3.4.13) from bullseye that you prepared in T418915#11851872 for bookworm as well? That would be the last moving part of this intermediate in-place step to bookworm before stepping to trixie.

Running a half marathon is always easier than a full one :-) I don't have time for it today, but will take care of a backport tomorrow.

Thank you very much, @MoritzMuehlenhoff. Other than compatibility tests I'll run in the interim, we're unlikely to use the new packages for at least two weeks, so take your time.

Change #1301364 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Add component/zookeeper34

https://gerrit.wikimedia.org/r/1301364

In T428495#12009735, @Scott_French wrote:

Thank you very much, @MoritzMuehlenhoff. Other than compatibility tests I'll run in the interim, we're unlikely to use the new packages for at least two weeks, so take your time.

I have a backport ready, patch for apt.wikimedia.org is at https://gerrit.wikimedia.org/r/1301364

Change #1301364 merged by Muehlenhoff:

[operations/puppet@production] Add component/zookeeper34

https://gerrit.wikimedia.org/r/1301364

Mentioned in SAL (#wikimedia-operations) [2026-06-15T06:53:16Z] <moritzm> imported zookeeper 3.4.13-6+wmf12u1 to component/zookeeper34 for bookworm-wikimedia T428495

Thank you very much, Moritz! Just tested it out in a mixed bullseye / bookworm cluster and everything seems to work as expected.

Change #1309257 had a related patch set uploaded (by Scott French; author: Scott French):

[operations/puppet@production] zookeeper: Support installation from component/zookeeper34

https://gerrit.wikimedia.org/r/1309257

Change #1309260 had a related patch set uploaded (by Scott French; author: Scott French):

[operations/puppet@production] hieradata: Use zookeeper34 forward-port component in codfw

https://gerrit.wikimedia.org/r/1309260

Change #1309315 had a related patch set uploaded (by Scott French; author: Scott French):

[operations/puppet@production] P:etcd::v3: Add support for explicitly enabling the v2 API

https://gerrit.wikimedia.org/r/1309315

Change #1309317 had a related patch set uploaded (by Scott French; author: Scott French):

[operations/puppet@production] hieradata: Explicitly enable the v2 API in codfw

https://gerrit.wikimedia.org/r/1309317

I've built etcd-mirror 0.0.12 packages for bullseye, bookworm, and trixie, as part of the work in T418915: conf200[7-9] implementation tracking. I've also tested all three versions in a simple docker compose testbed (with various permutations of up- and downstream etcd versions).

On Monday, I'll include the new packages and merge the remaining patches tagged on this task, at which point I believe all of the pre-work is complete.

What about moving to per-rack VLANs?

So, normally, we would take this (reimaging) as an opportunity to move hosts to the new per-rack VLANs. All three existing codfw hosts are eligible (T428191) as is conf1009 in eqiad (T421711).

At this time, I am going to propose we do not do this for any of the hosts in scope for now. Rationale:

1. The codfw hosts will be superseded in very short order by conf200[7-9], which are already on the new VLAN.
2. The sole eqiad host (conf1009) is eligible, but (a) there are some significant complications around Zookeeper IP changes and (b) we intend to refresh all three eqiad hosts in this FY.

A bit more detail on my concerns about Zookeeper:

At least as of the version of Zookeeper we are using (3.4), which also does not yet support dynamic reconfiguration, there is a strong likelihood that Zookeeper will not re-resolve the hostname of a peer when attempting to reconnect to it (e.g., when the peer returns from reimage / VLAN-move) and must be restarted in order to observe the IP change [0].

While that should generally be OK, there is a problem: Our Zookeeper clusters are 3 hosts. If one host is unreachable due to an as-yet unobserved IP change, restarting one more means we transiently loose quorum (i.e., potentially impactful for clients). I need to understand this much better if we're going to pursue this, and IMO it's a higher priority to get upgraded (particularly given 2.b above).

Note that an IP change isn't really a problem with etcd, thanks to what runtime reconfiguration allows us to do.

[0] For example, https://wikitech.wikimedia.org/wiki/Zookeeper#Changing_Zookeeper_IP_addresses mentions this, though that is clearly rather old.