In T428573#12023688, @CDanis wrote:

It might be possible to work around this by creating headless services for these jobs that access the databases -- definitely worth trying.

@Clement_Goubert @JMeybohm Does this sound crazypants to you?

In T423852#12018485, @MLechvien-WMF wrote:

@JMeybohm is this a hard dependency of T427400: Update calico to 3.30 (and in turn of the Kubernetes upgrade)?

In T423251#12009217, @Blake wrote:

charts/jaeger/charts/common/templates/_capabilities.tpl (min-version check, not sure if anything is required here)
charts/ipoid/Chart.yaml (min-version)
charts/benthos/Chart.yaml (min-version)

Linking T386694: Replace k8s-controller-sidecars with built in Sidecar containers on k8s 1.31 here since it's adjacent/similar work

In T428654#12001627, @Jhancock.wm wrote:

unplugged the PSUs and drained the flea power. had it PXE boot at Jayme's request. Reboot fixed it, but if it happens again, please let us know and we'll work with Dell to find the cause and replace it.

In T427401#11993792, @MLechvien-WMF wrote:

Will this patch {T423685} too?

Since we already started planning the next k8s upgrade (T42706: Fatal error in Special:AdvancedTranslate for special pages aliases) could you please prioritize this? Unfortunately upstream seems to always only support three k8s versions with each release, so we can't cover both.

AIUI the migration has been completed with Iab94dc

Moving to scheduled since we'll be doing some of those this Q

The fix from T428078: Swap partition not used when reimaging to trixie with reuse-parts.cfg and mdadm + lvm works fine so we may continue with the reimages as planned @jasmine_

Reimaging kafka-main2007.codfw.wmnet with this change worked just fine

With the two workarounds the installer does on longer hang on asking for swap. Unfortunately it still failed on first try to create a filesystem for the root partition (without saying so ofc). Restarting the reimage did work but it did not include the swap partition in /etc/fstab (and did not activate it).
I will complete the reimage so we have the broker back and it can catch up with what it missed (since /srv seems to not have been formatted, yay) but before taking on the next broker we should figure out what's happening/not happening here.

The patch might be not enough (will test anyways). If not, we might need to add the workaround from T408777: Swap partition issue when installing a DB with Debian Trixie

In T427088#11973521, @ops-monitoring-bot wrote:

Cookbook cookbooks.sre.hosts.reimage started by jasmine@cumin2002 for host kafka-main2006.codfw.wmnet with OS trixie executed with errors:

Just a blib

I totally forgot that debdeploy works on source packages, sorry!

@DPogorzelski-WMF all these[1] changelog entries have wrongly formatted footers (maintainer name and email, see https://manpages.debian.org/testing/dpkg-dev/deb-changelog.5.en.html) which leads to multiple warnings from docker-pkg. Would you mind fixing please?

Deployed to all clusters

I've deployed the version 2.14 to all thee staging clusters, will to the prod deploys EU tomorrow.

I don't see anything concerning in the changelogs from 2.10 to 2.14. Deprecated kube_endpoint_address_* metrics have been removed, but I could not find users of those across alerts repo and grafana.

This has been done in T414417: Add support for k8s 1.31 on trixie, https://gerrit.wikimedia.org/r/c/1240725