Change #1248108 had a related patch set uploaded (by Scott French; author: Scott French):

[operations/puppet@production] Add new conf200[789] hosts

https://gerrit.wikimedia.org/r/1248108

Change #1248108 merged by Scott French:

[operations/puppet@production] Add new conf200[789] hosts

https://gerrit.wikimedia.org/r/1248108

Wouldn't it make sense to directly move to trixie and skip bookworm?

@MoritzMuehlenhoff - So, there were two motivations for targeting bookworm initially:

1. Concerns about incompatibility between etcd 3.3 (bullseye) and 3.5 (trixie) - Skipping a minor version isn't a supported upgrade path, and we wanted to preserve our ability to incrementally reimage (vs. bootstrapping a new cluster from a blank slate).
2. Some questions about compatibility of profile::etcd::v3 on trixie. These have been resolved with recent work to reimage k8s control-plane nodes to trixie.

So, at this point, the only etcd-specific hangup is #1. Bootstrapping a new cluster isn't a deal-breaker for this particular etcd use case: We would shunt all traffic to the eqiad cluster during the work anyway, and can easily backfill data with etcd-mirror. Which is to say, targeting trixie may very well be fine, and indeed would save us an additional round of reimages and associated ceremony.

I don't know off hand if there might be additional questions about Zookeeper compatibility as well ... From a quick check, it looks like we'd be going from 3.4.x to either 3.8.x (bookworm) or 3.9.x (trixie), so per the backward compatibility rules I believe we're in for pain either way.

Anyway, long-winded way of saying: We can probably reassess and switch gears to trixie. I'll give some thought in the interim to what the bootstrapping procedure would look like.

we're having the issue that was documented in https://phabricator.wikimedia.org/T418929 with these servers. still working on a solution.

@Jhancock.wm Hi! I am testing a new version of the provision cookbook to unblock this use case, do you mind to send to me the BMC's passwords via email or IRC?

@elukey sent you the email

I provisioned the host, I see the following in reimage:

┌───────────────────────┤ [!!] Partition disks ├────────────────────────┐    
│                                                                       │    
│                 Failed to partition the selected disk                 │    
│ This probably happened because the selected disk or free space is too │    
│ small to be automatically partitioned.

These are the disks that d-i sees:

~ # ls /dev/sd*
/dev/sda   /dev/sda2  /dev/sdb   /dev/sdb2
/dev/sda1  /dev/sda3  /dev/sdb1  /dev/sdb3

But the recipe in preseed.yaml seems wrong, we need raid1-2dev-efi.cfg. Going to file a patch :)

Change #1302921 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] preseed: fix partman config for the new conf2* hosts

https://gerrit.wikimedia.org/r/1302921

Change #1302921 merged by Elukey:

[operations/puppet@production] preseed: fix partman config for the new conf2* hosts

https://gerrit.wikimedia.org/r/1302921

All hosts provisioned and reimaged.

Please keep in mind that I used test-cookbook to test new changes, they are still in code review so you cannot re-run reimage/provision on these nodes yet :D

Amazing - thank you very much, @elukey! (and duly noted about the unmerged cookbook changes)