⚓ T401832 Upgrade Traffic hosts to trixie

Subject	Repo	Branch	Lines +/-
varnishkafka: Only enable for text	operations/puppet	production	+6 -16
ats: Set secondary nvme drives for new codfw hosts	operations/puppet	production	+80 -0
hieradata: Set HAProxy version to 3 for cp204[34]	operations/puppet	production	+2 -0
Version 0.4.6~deb13u1	operations/debs/python-logstash	master	+6 -0
Also install the pbuilder hooks for trixie	operations/puppet	production	+6 -6
cache::haproxy: Only use lua5.3 mmdb on haproxy28	operations/puppet	production	+7 -2
cache::dp_key: Fix cross-filesystems tempfile error on Trixie	operations/puppet	production	+2 -1
cache::wmfuniq: Fix cross-filesystem tempfile error on Trixie	operations/puppet	production	+4 -1
varnish:common: Re-add buster version for Python	operations/puppet	production	+1 -0
varnish:common: add trixie version for Python	operations/puppet	production	+3 -3
site.pp: Provision cp2043/cp2044 as cache nodes	operations/puppet	production	+3 -3
aptrepo,haproxy: Allow HAProxy 3.0 on bullseye	operations/puppet	production	+12 -1
Release 0.9.8-1+wmf13u1	operations/debs/python-anycast-healthchecker	master	+8 -1

Status	Assigned	Task
In Progress	None	T401832 Upgrade Traffic hosts to trixie
Open	Vgutierrez	T401824 Get ready to upgrade liberica LBs to trixie
In Progress	CDobbins	T381608 Upgrade pdns-recursor to 5.x on all prod DNS hosts (all C:dnsrecursor and so possibly WMCS)
Open	CDobbins	T389333 Migrate PDNS recursor config to use /etc/powerdns/recursor.d ?
Open	None	T407320 Package benthos/redpanda for trixie
Open	None	T407532 Build Apache Trafficserver for Trixie
Resolved	Fabfur	T417253 Upgrade to HAProxy 3.0 on cache (bullseye) hosts
Open	None	T417291 Upgrade the CDN to HAProxy 3.0
Resolved	SLyngshede-WMF	T418161 Reimage cp20[43-58] to Trixie

Cookbook cookbooks.sre.hosts.reimage started by slyngshede@cumin1003 for host cp2043.codfw.wmnet with OS trixie completed:

cp2043 (PASS)
- Downtimed on Icinga/Alertmanager
- Disabled Puppet
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced UEFI HTTP Boot for next reboot
- Host rebooted via Redfish
- Host up (Debian installer)
- Host up (new fresh trixie OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- Removed previous downtime on Alertmanager (old OS)
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202602111352_slyngshede_2859730_cp2043.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- Updated Netbox data from PuppetDB

Some clarification, too: T386796#10642779 suggests that 3.1 was problematic. The hope is that 3.0 will not contain those issues and that the upgrade to 3.2 (or 3.3) will skip those issues.

@Vgutierrez / @Fabfur Regarding upgrading from 3.0 in Bullseye to 3.0 in Trixie.... 3.0 in Bullseye (via haproxy.debian.net) would pull in the latest 3.0.15... but then upgrading to Trixie and utilizing the repo's package would mean that we switch to 3.0.11-1+deb13u1. When upgrading to 3.0 in Bullseye, I propose we instead use the version 3.0.11-1~bpo11+1. It appears that the codebase would be the same for both distributions.

Fabfur created subtask T417253: Upgrade to HAProxy 3.0 on cache (bullseye) hosts.Thu, Feb 12, 9:43 AM

Fabfur updated the task description. (Show Details)Thu, Feb 12, 9:50 AM

Change #1238403 merged by Vgutierrez:

[operations/puppet@production] aptrepo,haproxy: Allow HAProxy 3.0 on bullseye

https://gerrit.wikimedia.org/r/1238403

Mentioned in SAL (#wikimedia-operations) [2026-02-12T15:23:29Z] <vgutierrez> fetch haproxy 3.0.15 on thirdparty/haproxy30 (bullseye-wikimedia) - T401832

Maintenance_bot removed a project: Patch-For-Review.Thu, Feb 12, 3:30 PM

Cookbook cookbooks.sre.hosts.reimage was started by sukhe@cumin1003 for host durum3005.esams.wmnet with OS trixie

Cookbook cookbooks.sre.hosts.reimage was started by sukhe@cumin1003 for host durum7004.magru.wmnet with OS trixie

Cookbook cookbooks.sre.hosts.reimage started by sukhe@cumin1003 for host durum3005.esams.wmnet with OS trixie completed:

durum3005 (PASS)
- Downtimed on Icinga/Alertmanager
- Disabled Puppet
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via gnt-instance
- Host up (Debian installer)
- Set boot media to disk
- Host up (new fresh trixie OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- Removed previous downtime on Alertmanager (old OS)
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202602121614_sukhe_3520396_durum3005.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- Updated Netbox data from PuppetDB

Cookbook cookbooks.sre.hosts.reimage started by sukhe@cumin1003 for host durum7004.magru.wmnet with OS trixie completed:

durum7004 (PASS)
- Downtimed on Icinga/Alertmanager
- Disabled Puppet
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via gnt-instance
- Host up (Debian installer)
- Set boot media to disk
- Host up (new fresh trixie OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- Removed previous downtime on Alertmanager (old OS)
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202602121629_sukhe_3532274_durum7004.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- Updated Netbox data from PuppetDB

Cookbook cookbooks.sre.hosts.reimage was started by sukhe@cumin1003 for host durum7003.magru.wmnet with OS trixie

Cookbook cookbooks.sre.hosts.reimage was started by sukhe@cumin1003 for host durum3006.esams.wmnet with OS trixie

Cookbook cookbooks.sre.hosts.reimage started by sukhe@cumin1003 for host durum3006.esams.wmnet with OS trixie completed:

durum3006 (PASS)
- Downtimed on Icinga/Alertmanager
- Disabled Puppet
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via gnt-instance
- Host up (Debian installer)
- Set boot media to disk
- Host up (new fresh trixie OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- Removed previous downtime on Alertmanager (old OS)
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202602121723_sukhe_3592573_durum3006.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- Updated Netbox data from PuppetDB

Cookbook cookbooks.sre.hosts.reimage started by sukhe@cumin1003 for host durum7003.magru.wmnet with OS trixie completed:

durum7003 (PASS)
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via gnt-instance
- Host up (Debian installer)
- Set boot media to disk
- Host up (new fresh trixie OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202602121727_sukhe_3591897_durum7003.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- Updated Netbox data from PuppetDB

All durum hosts upgraded to trixie, timestamping.

@SLyngshede-WMF If you could get a trixie host up and running in codfw with some of the new hardware that'd give us a great launchpad for testing out a trixie host.

In the meantime, @CDobbins and I are going to try and get a horizon trixie host up and running.

ssingh updated the task description. (Show Details)Thu, Feb 12, 6:37 PM

@BCornwall We currently have two hosts:

cp2043
cp2044

Both are now on Trixie

Change #1239383 had a related patch set uploaded (by BCornwall; author: BCornwall):

[operations/puppet@production] site.pp: Provision cp2043/cp2044 as cache nodes

https://gerrit.wikimedia.org/r/1239383

gerritbot added a project: Patch-For-Review.Fri, Feb 13, 6:07 PM

Change #1239383 merged by BCornwall:

[operations/puppet@production] site.pp: Provision cp2043/cp2044 as cache nodes

https://gerrit.wikimedia.org/r/1239383

Change #1239401 had a related patch set uploaded (by BCornwall; author: BCornwall):

[operations/puppet@production] varnish:common: add trixie version for Python

https://gerrit.wikimedia.org/r/1239401

Change #1239401 merged by BCornwall:

[operations/puppet@production] varnish:common: add trixie version for Python

https://gerrit.wikimedia.org/r/1239401

Maintenance_bot removed a project: Patch-For-Review.Fri, Feb 13, 10:31 PM

Change #1239434 had a related patch set uploaded (by BCornwall; author: BCornwall):

[operations/puppet@production] varnish:common: Re-add buster version for Python

https://gerrit.wikimedia.org/r/1239434

gerritbot added a project: Patch-For-Review.Fri, Feb 13, 10:41 PM

Change #1239434 merged by BCornwall:

[operations/puppet@production] varnish:common: Re-add buster version for Python

https://gerrit.wikimedia.org/r/1239434

Mentioned in SAL (#wikimedia-operations) [2026-02-13T23:01:44Z] <brett> Import varnishkafka 1.2.0~deb13+wmf1 into trixie-wikimedia (T401832)

BCornwall updated the task description. (Show Details)Fri, Feb 13, 11:03 PM

Mentioned in SAL (#wikimedia-operations) [2026-02-13T23:25:36Z] <brett> Import prometheus-varnishkafka-exporter 0.1~deb13u1 into trixie-wikimedia (T401832)

Maintenance_bot removed a project: Patch-For-Review.Fri, Feb 13, 11:31 PM

Change #1239460 had a related patch set uploaded (by BCornwall; author: BCornwall):

[operations/debs/python-logstash@master] Version 0.4.6~deb13u1

https://gerrit.wikimedia.org/r/1239460

gerritbot added a project: Patch-For-Review.Sat, Feb 14, 2:45 AM

BCornwall updated the task description. (Show Details)Sat, Feb 14, 3:07 AM

@Fabfur I uploaded a new trixie-wikimedia branch on lua-maxminddb's repo - sadly, I can't seem to create an MR into a non-existent branch so I just created the branch. Could you check it and approve the commit/packaging? :)

Mentioned in SAL (#wikimedia-operations) [2026-02-14T03:37:30Z] <brett> Import lua5.4-maxminddb 0.1.1~deb13u1 into trixie-wikimedia (T401832)

BCornwall updated the task description. (Show Details)Sat, Feb 14, 3:40 AM

Change #1239463 had a related patch set uploaded (by BCornwall; author: BCornwall):

[operations/puppet@production] cache::haproxy: Only use lua5.3 mmdb on

https://gerrit.wikimedia.org/r/1239463

Vgutierrez added a subtask: T417291: Upgrade the CDN to HAProxy 3.0.Mon, Feb 16, 9:21 AM

Change #1239602 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/puppet@production] cache::wmfuniq: Use same filesystem for tempfile to avoid cross-filesystem errors

https://gerrit.wikimedia.org/r/1239602

Change #1239602 merged by Vgutierrez:

[operations/puppet@production] cache::wmfuniq: Fix cross-filesystem tempfile error on Trixie

https://gerrit.wikimedia.org/r/1239602

Change #1239631 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/puppet@production] cache::dp_key: Fix cross-filesystems tempfile error on Trixie

https://gerrit.wikimedia.org/r/1239631

Change #1239631 merged by Vgutierrez:

[operations/puppet@production] cache::dp_key: Fix cross-filesystems tempfile error on Trixie

https://gerrit.wikimedia.org/r/1239631

Vgutierrez updated the task description. (Show Details)Mon, Feb 16, 3:19 PM

Change #1239463 abandoned by BCornwall:

[operations/puppet@production] cache::haproxy: Only use lua5.3 mmdb on haproxy28

Reason:

Idee228cf20c011040052d9e4e6e1349de94893f9

https://gerrit.wikimedia.org/r/1239463

Mentioned in SAL (#wikimedia-operations) [2026-02-17T14:07:03Z] <vgutierrez> upload golang-github-florianl-go-tc_0.4.7 to trixie-wikimedia (apt.wm.o) - T401832

Mentioned in SAL (#wikimedia-operations) [2026-02-17T14:58:25Z] <vgutierrez> upload golang-github-mmatczuk-anyflag-dev 0.0~git20240709.eb9e24c-1 to trixie-wikimedia (apt.wm.o) - T401832

Change #1239963 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Also install the pbuilder hooks for trixie

https://gerrit.wikimedia.org/r/1239963

Change #1239963 merged by Muehlenhoff:

[operations/puppet@production] Also install the pbuilder hooks for trixie

https://gerrit.wikimedia.org/r/1239963

Change #1239460 abandoned by BCornwall:

[operations/debs/python-logstash@master] Version 0.4.6~deb13u1

Reason:

Moved to gitlab: https://gitlab.wikimedia.org/repos/sre/python-logstash

https://gerrit.wikimedia.org/r/1239460

Mentioned in SAL (#wikimedia-operations) [2026-02-17T17:57:00Z] <brett> Import python-logstash (python3-logstash) 0.4.6~deb13u1 to trixie-wikimedia (T401832)

BCornwall updated the task description. (Show Details)Tue, Feb 17, 5:57 PM

BCornwall updated the task description. (Show Details)Tue, Feb 17, 6:03 PM

Maintenance_bot removed a project: Patch-For-Review.Tue, Feb 17, 6:30 PM

CDobbins updated the task description. (Show Details)Tue, Feb 17, 7:37 PM

Change #1240064 had a related patch set uploaded (by BCornwall; author: BCornwall):

[operations/puppet@production] hieradata: Set HAProxy version to 3 for cp204[34]

https://gerrit.wikimedia.org/r/1240064

gerritbot added a project: Patch-For-Review.Tue, Feb 17, 9:43 PM

Change #1240064 merged by BCornwall:

[operations/puppet@production] hieradata: Set HAProxy version to 3 for cp204[34]

https://gerrit.wikimedia.org/r/1240064

BCornwall updated the task description. (Show Details)Tue, Feb 17, 10:01 PM

Maintenance_bot removed a project: Patch-For-Review.Tue, Feb 17, 10:31 PM

Mentioned in SAL (#wikimedia-operations) [2026-02-18T11:57:48Z] <vgutierrez> upload golang-github-intel-go-cpuid 0.0~git20210602.5747e5c-2+deb13u1 to trixie-wikimedia (apt.wm.o) - T401832

Mentioned in SAL (#wikimedia-operations) [2026-02-18T14:39:19Z] <vgutierrez> upload golang-github-u-root-u-root 0.12.0-1 to trixie-wikimedia (apt.wm.o) - T401832

Mentioned in SAL (#wikimedia-operations) [2026-02-18T14:48:10Z] <vgutierrez> upload golang-gitlab-wikimedia-sre-qemutest-dev 0.1.0+deb13u1 to trixie-wikimedia (apt.wm.o) - T401832

Mentioned in SAL (#wikimedia-operations) [2026-02-18T14:54:12Z] <vgutierrez> uplodaded tcp-mss-clamper 0.6+deb13u1 to trixie-wikimedia (apt-wm.o) - T401832

Vgutierrez updated the task description. (Show Details)Wed, Feb 18, 2:54 PM

Mentioned in SAL (#wikimedia-operations) [2026-02-18T19:05:51Z] <brett> import haproxykafka 0.3.16+deb13u1 into trixie-wikimedia (T401832)

BCornwall updated the task description. (Show Details)Wed, Feb 18, 7:49 PM

BCornwall added a subtask: T418161: Reimage cp20[43-58] to Trixie.Mon, Feb 23, 5:49 PM

BCornwall changed the status of subtask T418161: Reimage cp20[43-58] to Trixie from Open to In Progress.

Change #1243184 had a related patch set uploaded (by BCornwall; author: BCornwall):

[operations/puppet@production] ats: Set secondary nvme drives for new codfw hosts

https://gerrit.wikimedia.org/r/1243184

gerritbot added a project: Patch-For-Review.Tue, Feb 24, 5:36 PM

Change #1243195 had a related patch set uploaded (by BCornwall; author: BCornwall):

[operations/puppet@production] varnishkafka: Only enable prom exporter for text

https://gerrit.wikimedia.org/r/1243195

Mentioned in SAL (#wikimedia-operations) [2026-02-24T22:37:20Z] <brett> import ncmonitor 3.1.0~deb13u1 into trixie-wikimedia (T401832)

Cookbook cookbooks.sre.hosts.reimage was started by brett@cumin2002 for host ncmonitor1001.eqiad.wmnet with OS trixie

BCornwall updated the task description. (Show Details)Tue, Feb 24, 11:10 PM

Cookbook cookbooks.sre.hosts.reimage started by brett@cumin2002 for host ncmonitor1001.eqiad.wmnet with OS trixie completed:

ncmonitor1001 (PASS)
- Downtimed on Icinga/Alertmanager
- Disabled Puppet
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via gnt-instance
- Host up (Debian installer)
- Set boot media to disk
- Host up (new fresh trixie OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- Removed previous downtime on Alertmanager (old OS)
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202602242259_brett_2931472_ncmonitor1001.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- Updated Netbox data from PuppetDB

BCornwall updated the task description. (Show Details)Tue, Feb 24, 11:50 PM

Change #1243184 merged by BCornwall:

[operations/puppet@production] ats: Set secondary nvme drives for new codfw hosts

https://gerrit.wikimedia.org/r/1243184

Cookbook cookbooks.sre.hosts.reimage was started by cdobbins@cumin2002 for host cp2043.codfw.wmnet with OS trixie

Cookbook cookbooks.sre.hosts.reimage started by cdobbins@cumin2002 for host cp2043.codfw.wmnet with OS trixie completed:

cp2043 (PASS)
- Downtimed on Icinga/Alertmanager
- Set pooled=inactive for the following services on confctl:

{"cp2043.codfw.wmnet": {"weight": 0, "pooled": "inactive"}, "tags": "dc=codfw,cluster=cache_text,service=cdn"}
{"cp2043.codfw.wmnet": {"weight": 0, "pooled": "inactive"}, "tags": "dc=codfw,cluster=cache_text,service=ats-be"}

Disabled Puppet
Removed from Puppet and PuppetDB if present and deleted any certificates
Removed from Debmonitor if present
Forced UEFI HTTP Boot for next reboot
Host rebooted via Redfish
Host up (Debian installer)
Host up (new fresh trixie OS)
Generated Puppet certificate
Signed new Puppet certificate
Run Puppet in NOOP mode to populate exported resources in PuppetDB
Found Nagios_host resource for this host in PuppetDB
Downtimed the new host on Icinga/Alertmanager
Removed previous downtime on Alertmanager (old OS)
First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202602252048_cdobbins_3572558_cp2043.out
configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
Rebooted
Automatic Puppet run was successful
Forced a re-check of all Icinga services for the host
Icinga status is optimal
Icinga downtime removed
No changes in confctl are needed to restore the previous state.
Updated Netbox data from PuppetDB

Cookbook cookbooks.sre.hosts.reimage was started by cdobbins@cumin2002 for host cp2044.codfw.wmnet with OS trixie

Cookbook cookbooks.sre.hosts.reimage started by cdobbins@cumin2002 for host cp2044.codfw.wmnet with OS trixie completed:

cp2044 (WARN)
- Downtimed on Icinga/Alertmanager
- Set pooled=inactive for the following services on confctl:

{"cp2044.codfw.wmnet": {"weight": 0, "pooled": "no"}, "tags": "dc=codfw,cluster=cache_upload,service=ats-be"}
{"cp2044.codfw.wmnet": {"weight": 0, "pooled": "no"}, "tags": "dc=codfw,cluster=cache_upload,service=cdn"}

Disabled Puppet
Removed from Puppet and PuppetDB if present and deleted any certificates
Removed from Debmonitor if present
Forced UEFI HTTP Boot for next reboot
Host rebooted via Redfish
Host up (Debian installer)
Host up (new fresh trixie OS)
Generated Puppet certificate
Signed new Puppet certificate
Run Puppet in NOOP mode to populate exported resources in PuppetDB
Found Nagios_host resource for this host in PuppetDB
Downtimed the new host on Icinga/Alertmanager
Removed previous downtime on Alertmanager (old OS)
First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202602252135_cdobbins_3606681_cp2044.out
configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
Rebooted
Automatic Puppet run was successful
Forced a re-check of all Icinga services for the host
Icinga status is optimal
Icinga downtime removed
Some services have a zero weight, you have to set a weight with:

sudo confctl select '{tags_line}' set/weight=NN
sudo confctl select '{tags_line}' set/weight=NN

Services in confctl are not automatically pooled, to restore the previous state you have to run the following commands:

sudo confctl select 'name=cp2044\.codfw\.wmnet,dc=codfw,cluster=cache_upload,service=ats\-be' set/pooled=no
sudo confctl select 'name=cp2044\.codfw\.wmnet,dc=codfw,cluster=cache_upload,service=cdn' set/pooled=no

Updated Netbox data from PuppetDB

Change #1243195 merged by BCornwall:

[operations/puppet@production] varnishkafka: Only enable for text

https://gerrit.wikimedia.org/r/1243195

Maintenance_bot removed a project: Patch-For-Review.Thu, Feb 26, 6:31 PM

Cookbook cookbooks.sre.hosts.reimage was started by cdobbins@cumin2002 for host cp2047.codfw.wmnet with OS trixie

Cookbook cookbooks.sre.hosts.reimage started by cdobbins@cumin2002 for host cp2047.codfw.wmnet with OS trixie completed:

cp2047 (PASS)
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced UEFI HTTP Boot for next reboot
- Host rebooted via Redfish
- Host up (Debian installer)
- Host up (new fresh trixie OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202602262102_cdobbins_127988_cp2047.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- Updated Netbox data from PuppetDB

Cookbook cookbooks.sre.hosts.reimage was started by cdobbins@cumin2002 for host cp2057.codfw.wmnet with OS trixie

Cookbook cookbooks.sre.hosts.reimage was started by cdobbins@cumin2002 for host cp2058.codfw.wmnet with OS trixie

Cookbook cookbooks.sre.hosts.reimage started by cdobbins@cumin2002 for host cp2057.codfw.wmnet with OS trixie completed:

cp2057 (PASS)
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced UEFI HTTP Boot for next reboot
- Host rebooted via Redfish
- Host up (Debian installer)
- Host up (new fresh trixie OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202602272114_cdobbins_866779_cp2057.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- Updated Netbox data from PuppetDB

Cookbook cookbooks.sre.hosts.reimage started by cdobbins@cumin2002 for host cp2058.codfw.wmnet with OS trixie completed:

cp2058 (PASS)
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced UEFI HTTP Boot for next reboot
- Host rebooted via Redfish
- Host up (Debian installer)
- Host up (new fresh trixie OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202602272117_cdobbins_870415_cp2058.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- Updated Netbox data from PuppetDB

CDobbins closed subtask T418161: Reimage cp20[43-58] to Trixie as Resolved.Fri, Feb 27, 10:26 PM

BCornwall updated the task description. (Show Details)Wed, Mar 4, 7:07 PM

Vgutierrez closed subtask T417253: Upgrade to HAProxy 3.0 on cache (bullseye) hosts as Resolved.Fri, Mar 6, 4:36 PM

Upgrade Traffic hosts to trixie
Open, In Progress, LowPublic
Actions

Description

Debian Packaging

A:acmechief

A:cp

A:dnsbox

A:durum

A:lvs

A:ncmonitor

A:ncredir

A:wikidough

trixie reimaging

A:acmechief

A:cp

A:durum

A:ncmonitor

A:ncredir

A:dnsbox

A:wikidough

Details

Related Objects
Search...

Event Timeline

Host	Debian Packages	Reimaging
acmechief
cp	done
dns
durum	done	done
lvs
ncmonitor	done	done
ncredir
wikidough
haproxykafka

	ssingh
	Aug 13 2025, 2:19 PM

Upgrade Traffic hosts to trixieOpen, In Progress, LowPublicActions

Description

Debian Packaging

A:acmechief

A:cp

A:dnsbox

A:durum

A:lvs

A:ncmonitor

A:ncredir

A:wikidough

trixie reimaging

A:acmechief

A:cp

A:durum

A:ncmonitor

A:ncredir

A:dnsbox

A:wikidough

Details

Related ObjectsSearch...

Event Timeline

Upgrade Traffic hosts to trixie
Open, In Progress, LowPublic
Actions

Related Objects
Search...