It's depooled and monitoring disabled, you can replace any time

ops-limited is very broad access, it grants access to any of our 2400 server, including some very sensitive ones. But if this access is an ongoing need, we can surely create a new group kafka-jumbo-access, which grants you shell access to the Kafka Jumbo nodes.

The initial imposm catchup sync after the PBF import has just completed.

Thanks, I'll rebuild the software RAID tomorrow

The server has a defective memory stick, adding DC ops to get it deplaced:

RAID is rebuilt, resolving.

Yes, please. I'll take care of the software RAID rebuild.

This has been implemented. For installing a new server partman/custom/db-efi.cfg needs to be selected.

In T410195#11420424, @Jhancock.wm wrote:

@MoritzMuehlenhoff thanks for the help and correction. 22353BB15C0C has been replaced.

@Jhancock.wm The broken disk is /dev/sda which per lshw has the serial 22353BB15C0C, does that help?

Updates have been rolled out and diffs are being sent again.

Also reported to Debian as https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1121730

Rancid is a bit of a maze of scripts calling each other, but I could eventually track it down to /usr/bin/control_rancid. In our case, the key provided by "system tls server-profile self-signed key" is what made it too long eventaully for Exim to accept.

In T410743#11415797, @Jclark-ctr wrote:

Supermicro has been slow to respond to case updates, usually taking 2–3 days between replies, while I have been responding the same day. They finally agreed to ship the replacement parts today, but since they’re now asking me to confirm the address I already provided, it will probably be another 2–3 days before they acknowledge that email and actually ship the parts.

Access has been granted via Wikimedia IDM on Nov. 26, 2025, 4:38 p.m. Marking this task as resolved

I think all the relevant cookbooks have this enabled now. I'm resolving the task.

In T409832#11404160, @ABran-WMF wrote:

@MoritzMuehlenhoff as @Dzahn mentioned, you might be interested to review what's been done on the gitlab-package-puller script. I'd be happy to have a chat about it if needed, let me know if you see any missing things or any room for improvement

All done

These are all done, the remaining Ganeti nodes w/o AAAA records were decommissioned as part of the last hardware refreshes in eqiad and codfw.

This got fixed as part of the migration to Bookworm.

In T410573#11412023, @RKemper wrote:

Ah yeah the last couple batches were still ongoing. Should be all current now.

@Jclark-ctr : You should now be able to run smartctl, let me know if you run into any issues

@RKemper There's still an a missed host: cirrussearch2084 is marked as fixed, but on and old kernel and has an uptime of 215 days

This is complete. Luca and myself made a total of 122 commits to puppet.git (plus surely a few where me missed to tag the task) for:

In T410195#11380368, @Jhancock.wm wrote:

Is this a false alert? I'm not seeing any issues physically with the server or in the idrac.

If this drive does need to be replaced, could i get some dmesg errors to send to Dell? they've been giving me trouble getting bad drives replaced.

Copied the output of dmesg to this paste in case it's needed for the warranty case: https://phabricator.wikimedia.org/P85732

dmesg is full of I/O errors for dev/sdb, we should definitely get that drive replaced.

FWIW, the plan for eqiad sounds good to me

I'll have a look later the day

The SuperMicro hosts are somewhat special, for the Dells the following cookbook should handle the reprovision to UEFI mode:

In T410406#11400638, @bking wrote:

I'll grab it back and update the partman recipes. Keep in mind that these are very old Dells as opposed to brand-new SuperMicros. I'm not sure of y'all's success rate for the Dell + UEFI combo, but please let me know if there's another other special handling needed for these hosts.

This broke Puppet runs on the puppetservers:

You can simply confirm and continue, Puppet 7 is already enabled for wdqs1031 via the insetup::data_platform_ferm role in site.pp

There's a few more trixie cloud nodes, I took the liberty to expand the task accordingly.

In T410612#11393300, @Clement_Goubert wrote:

@KOfori Could you approve this ?

All done

@Volans made a copy of old the Spicerack/Cumin logs , they are available in /var/log/cumin100[12] on cumin1003 in case anyone needs them.

In T409860#11388216, @ssingh wrote:

Oh wow, thanks @MoritzMuehlenhoff! But what was the issue for my understanding?

@ssingh The hcaptcha-proxy VMs in magru are up and running

In T401832#11386096, @BCornwall wrote:

@MatthewVernon Looks like you were a maintainer of pcre3 in Debian before it was axed in Trixie. Sadly, we're in need of that package for trafficserver (the current patches for enabling pcre2 support are immature and we just want to continue using pcre until it's ready).

Specs look good