Page MenuHomePhabricator
Create Task
Maniphest T401832

Upgrade Traffic hosts to trixie
Open, In Progress, LowPublic
Actions

Description

This task tracks the upgrade of the Traffic hosts to trixie, affecting the services below and identified by their cumin aliases.

This is meant to be an umbrella task for all changes that will be part of this upgrade, such as the Debian packaging, Puppet changes, and the related testing include reimaging.

Progress:

HostDebian PackagesReimaging
acmechief
cpdonedone
dnsdone
durumdonedone
lvs
ncmonitordonedone
ncredir
wikidough

Debian Packaging

This includes packages that we build ourselves.

A:acmechief

A:cp

A:dnsbox

  • gdnsd

A:durum

  • anycast-healthchecker

A:lvs

  • liberica

A:ncmonitor

A:ncredir

  • benthos/redpanda

A:wikidough

  • anycast-healthchecker
  • dnsdist

trixie reimaging

A:acmechief

  • acmechief2002.codfw.wmnet
  • acmechief1002.eqiad.wmnet
  • acmechief-test2001.codfw.wmnet
  • acmechief-test1001.eqiad.wmnet

A:cp

  • cp1100.eqiad.wmnet
  • cp1101.eqiad.wmnet
  • cp1102.eqiad.wmnet
  • cp1103.eqiad.wmnet
  • cp1104.eqiad.wmnet
  • cp1105.eqiad.wmnet
  • cp1106.eqiad.wmnet
  • cp1107.eqiad.wmnet
  • cp1108.eqiad.wmnet
  • cp1109.eqiad.wmnet
  • cp1110.eqiad.wmnet
  • cp1111.eqiad.wmnet
  • cp1112.eqiad.wmnet
  • cp1113.eqiad.wmnet
  • cp1114.eqiad.wmnet
  • cp1115.eqiad.wmnet Hardware issues (T421007)
  • cp3066.esams.wmnet
  • cp3067.esams.wmnet
  • cp3068.esams.wmnet
  • cp3069.esams.wmnet
  • cp3070.esams.wmnet
  • cp3071.esams.wmnet
  • cp3072.esams.wmnet
  • cp3073.esams.wmnet
  • cp3074.esams.wmnet
  • cp3075.esams.wmnet
  • cp3076.esams.wmnet
  • cp3077.esams.wmnet
  • cp3078.esams.wmnet
  • cp3079.esams.wmnet
  • cp3080.esams.wmnet
  • cp3081.esams.wmnet
  • cp4037.ulsfo.wmnet
  • cp4038.ulsfo.wmnet
  • cp4039.ulsfo.wmnet
  • cp4040.ulsfo.wmnet
  • cp4041.ulsfo.wmnet
  • cp4042.ulsfo.wmnet
  • cp4043.ulsfo.wmnet
  • cp4044.ulsfo.wmnet
  • cp4045.ulsfo.wmnet
  • cp4046.ulsfo.wmnet
  • cp4047.ulsfo.wmnet
  • cp4048.ulsfo.wmnet
  • cp4049.ulsfo.wmnet
  • cp4050.ulsfo.wmnet
  • cp4051.ulsfo.wmnet
  • cp4052.ulsfo.wmnet
  • cp5017.eqsin.wmnet
  • cp5018.eqsin.wmnet
  • cp5019.eqsin.wmnet
  • cp5020.eqsin.wmnet
  • cp5021.eqsin.wmnet
  • cp5022.eqsin.wmnet Hardware issues (T414411)
  • cp5023.eqsin.wmnet
  • cp5024.eqsin.wmnet
  • cp5025.eqsin.wmnet
  • cp5026.eqsin.wmnet
  • cp5027.eqsin.wmnet
  • cp5028.eqsin.wmnet
  • cp5029.eqsin.wmnet
  • cp5030.eqsin.wmnet
  • cp5031.eqsin.wmnet
  • cp5032.eqsin.wmnet
  • cp6001.drmrs.wmnet
  • cp6002.drmrs.wmnet
  • cp6003.drmrs.wmnet
  • cp6004.drmrs.wmnet
  • cp6005.drmrs.wmnet
  • cp6006.drmrs.wmnet
  • cp6007.drmrs.wmnet
  • cp6008.drmrs.wmnet
  • cp6009.drmrs.wmnet
  • cp6010.drmrs.wmnet
  • cp6011.drmrs.wmnet
  • cp6012.drmrs.wmnet
  • cp6013.drmrs.wmnet
  • cp6014.drmrs.wmnet
  • cp6015.drmrs.wmnet
  • cp6016.drmrs.wmnet
  • cp7001.magru.wmnet
  • cp7002.magru.wmnet
  • cp7003.magru.wmnet
  • cp7004.magru.wmnet
  • cp7005.magru.wmnet
  • cp7006.magru.wmnet
  • cp7007.magru.wmnet
  • cp7008.magru.wmnet
  • cp7009.magru.wmnet
  • cp7010.magru.wmnet
  • cp7011.magru.wmnet
  • cp7012.magru.wmnet
  • cp7013.magru.wmnet
  • cp7014.magru.wmnet
  • cp7015.magru.wmnet
  • cp7016.magru.wmnet

A:durum

  • durum1001.eqiad.wmnet
  • durum1002.eqiad.wmnet
  • durum2001.codfw.wmnet
  • durum2002.codfw.wmnet
  • durum3005.esams.wmnet
  • durum3006.esams.wmnet
  • durum4001.ulsfo.wmnet
  • durum4002.ulsfo.wmnet
  • durum5001.eqsin.wmnet
  • durum5002.eqsin.wmnet
  • durum6001.drmrs.wmnet
  • durum6002.drmrs.wmnet
  • durum7003.magru.wmnet
  • durum7004.magru.wmnet

A:ncmonitor

  • ncmonitor1001.eqiad.wmnet

A:ncredir

  • ncredir1001.eqiad.wmnet
  • ncredir1002.eqiad.wmnet
  • ncredir2001.codfw.wmnet
  • ncredir2002.codfw.wmnet
  • ncredir3005.esams.wmnet
  • ncredir3006.esams.wmnet
  • ncredir4001.ulsfo.wmnet
  • ncredir4002.ulsfo.wmnet
  • ncredir5001.eqsin.wmnet
  • ncredir5002.eqsin.wmnet
  • ncredir6001.drmrs.wmnet
  • ncredir6002.drmrs.wmnet
  • ncredir7003.magru.wmnet
  • ncredir7004.magru.wmnet

A:dnsbox

  • dns1004.wikimedia.org
  • dns1005.wikimedia.org
  • dns1006.wikimedia.org
  • dns2004.wikimedia.org
  • dns2005.wikimedia.org
  • dns2006.wikimedia.org
  • dns3003.wikimedia.org
  • dns3004.wikimedia.org
  • dns4003.wikimedia.org
  • dns4004.wikimedia.org
  • dns5003.wikimedia.org
  • dns5004.wikimedia.org
  • dns6001.wikimedia.org
  • dns6002.wikimedia.org
  • dns7001.wikimedia.org
  • dns7002.wikimedia.org

A:wikidough

  • doh1001.wikimedia.org
  • doh1002.wikimedia.org
  • doh2001.wikimedia.org
  • doh2002.wikimedia.org
  • doh3005.wikimedia.org
  • doh3006.wikimedia.org
  • doh4001.wikimedia.org
  • doh4002.wikimedia.org
  • doh5001.wikimedia.org
  • doh5002.wikimedia.org
  • doh6001.wikimedia.org
  • doh6002.wikimedia.org
  • doh7003.wikimedia.org
  • doh7004.wikimedia.org

Details

Related Changes in Gerrit:
SubjectAuthorRepoBranchLines +/-
hieradata: issue fixesCDobbinsoperations/puppetproduction+20 -19
varnish: Update tests container image to TrixieSlyngshedeoperations/puppetproduction+3 -3
hieradata: remove pdns cfg override for dns7002CDobbinsoperations/puppetproduction+0 -1
hieradata: override dns7002; use correct cfg fileCDobbinsoperations/puppetproduction+1 -0
hieradata: override dns7002; use correct cfg fileCDobbinsoperations/puppetproduction+1 -1
dnsrecursor: add quotes to outgoing.dont_query valsCDobbinsoperations/puppetproduction+1 -1
dnsrecursor: fix file nameCDobbinsoperations/puppetproduction+2 -2
hiera: Set default codfw storage_elementsBCornwalloperations/puppetproduction+2 -81
hiera: Remove single_backend from codfwBCornwalloperations/puppetproduction+2 -20
varnishkafka: Only enable for textBCornwalloperations/puppetproduction+6 -16
ats: Set secondary nvme drives for new codfw hostsBCornwalloperations/puppetproduction+80 -0
hieradata: Set HAProxy version to 3 for cp204[34]BCornwalloperations/puppetproduction+2 -0
Version 0.4.6~deb13u1BCornwalloperations/debs/python-logstashmaster+6 -0
Also install the pbuilder hooks for trixieMuehlenhoffoperations/puppetproduction+6 -6
cache::haproxy: Only use lua5.3 mmdb on haproxy28BCornwalloperations/puppetproduction+7 -2
cache::dp_key: Fix cross-filesystems tempfile error on TrixieVgutierrezoperations/puppetproduction+2 -1
cache::wmfuniq: Fix cross-filesystem tempfile error on TrixieVgutierrezoperations/puppetproduction+4 -1
varnish:common: Re-add buster version for PythonBCornwalloperations/puppetproduction+1 -0
varnish:common: add trixie version for PythonBCornwalloperations/puppetproduction+3 -3
site.pp: Provision cp2043/cp2044 as cache nodesBCornwalloperations/puppetproduction+3 -3
aptrepo,haproxy: Allow HAProxy 3.0 on bullseyeBCornwalloperations/puppetproduction+12 -1
Release 0.9.8-1+wmf13u1Ssinghoperations/debs/python-anycast-healthcheckermaster+8 -1
Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
ops-monitoring-bot added a comment.Mar 18 2026, 3:51 PM

Cookbook cookbooks.sre.hosts.reimage started by cdobbins@cumin2002 for host cp3074.esams.wmnet with OS trixie completed:

  • cp3074 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh trixie OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202603181525_cdobbins_3740814_cp3074.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Mar 18 2026, 4:00 PM

Cookbook cookbooks.sre.hosts.reimage was started by cdobbins@cumin2002 for host cp3076.esams.wmnet with OS trixie

ops-monitoring-bot added a comment.Mar 18 2026, 4:00 PM

Cookbook cookbooks.sre.hosts.reimage started by cdobbins@cumin2002 for host cp3075.esams.wmnet with OS trixie completed:

  • cp3075 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh trixie OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202603181534_cdobbins_3740901_cp3075.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
CDobbins updated the task description. (Show Details)Mar 18 2026, 4:47 PM
ops-monitoring-bot added a comment.Mar 18 2026, 4:55 PM

Cookbook cookbooks.sre.hosts.reimage started by cdobbins@cumin2002 for host cp3076.esams.wmnet with OS trixie completed:

  • cp3076 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh trixie OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202603181629_cdobbins_3761299_cp3076.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Mar 18 2026, 5:01 PM

Cookbook cookbooks.sre.hosts.reimage was started by cdobbins@cumin2002 for host cp3077.esams.wmnet with OS trixie

ops-monitoring-bot added a comment.Mar 18 2026, 5:03 PM

Cookbook cookbooks.sre.hosts.reimage was started by cdobbins@cumin2002 for host cp3078.esams.wmnet with OS trixie

BCornwall updated the task description. (Show Details)Mar 18 2026, 5:08 PM
BCornwall updated the task description. (Show Details)
CDobbins updated the task description. (Show Details)Mar 18 2026, 5:16 PM
ops-monitoring-bot added a comment.Mar 18 2026, 5:56 PM

Cookbook cookbooks.sre.hosts.reimage started by cdobbins@cumin2002 for host cp3077.esams.wmnet with OS trixie completed:

  • cp3077 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh trixie OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202603181730_cdobbins_3781802_cp3077.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Mar 18 2026, 5:59 PM

Cookbook cookbooks.sre.hosts.reimage started by cdobbins@cumin2002 for host cp3078.esams.wmnet with OS trixie completed:

  • cp3078 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh trixie OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202603181733_cdobbins_3782074_cp3078.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
CDobbins updated the task description. (Show Details)Mar 18 2026, 6:13 PM
ops-monitoring-bot added a comment.Mar 18 2026, 6:17 PM

Cookbook cookbooks.sre.hosts.reimage was started by cdobbins@cumin2002 for host cp5018.eqsin.wmnet with OS trixie

ops-monitoring-bot added a comment.Mar 18 2026, 6:18 PM

Cookbook cookbooks.sre.hosts.reimage was started by cdobbins@cumin2002 for host cp5017.eqsin.wmnet with OS trixie

BCornwall updated the task description. (Show Details)Mar 18 2026, 7:09 PM
ops-monitoring-bot added a comment.Mar 18 2026, 7:26 PM

Cookbook cookbooks.sre.hosts.reimage started by cdobbins@cumin2002 for host cp5018.eqsin.wmnet with OS trixie completed:

  • cp5018 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh trixie OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202603181854_cdobbins_3801314_cp5018.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Mar 18 2026, 7:29 PM

Cookbook cookbooks.sre.hosts.reimage was started by cdobbins@cumin2002 for host cp5020.eqsin.wmnet with OS trixie

CDobbins updated the task description. (Show Details)Mar 18 2026, 7:36 PM
ops-monitoring-bot added a comment.Mar 18 2026, 7:39 PM

Cookbook cookbooks.sre.hosts.reimage started by cdobbins@cumin2002 for host cp5017.eqsin.wmnet with OS trixie executed with errors:

  • cp5017 (FAIL)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console cp5017.eqsin.wmnet" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.Mar 18 2026, 8:05 PM

Cookbook cookbooks.sre.hosts.reimage was started by cdobbins@cumin2002 for host cp5017.eqsin.wmnet with OS trixie

BCornwall updated the task description. (Show Details)Mar 18 2026, 8:23 PM
ops-monitoring-bot added a comment.Mar 18 2026, 8:50 PM

Cookbook cookbooks.sre.hosts.reimage started by cdobbins@cumin2002 for host cp5020.eqsin.wmnet with OS trixie executed with errors:

  • cp5020 (FAIL)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console cp5020.eqsin.wmnet" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.Mar 18 2026, 9:16 PM

Cookbook cookbooks.sre.hosts.reimage started by cdobbins@cumin2002 for host cp5017.eqsin.wmnet with OS trixie completed:

  • cp5017 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh trixie OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202603182044_cdobbins_3833393_cp5017.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
BCornwall updated the task description. (Show Details)Mar 18 2026, 11:05 PM
BCornwall updated the task description. (Show Details)
CDobbins updated the task description. (Show Details)Mar 19 2026, 3:37 PM
BCornwall updated the task description. (Show Details)Mar 19 2026, 5:07 PM
BCornwall updated the task description. (Show Details)Mar 19 2026, 5:38 PM
ops-monitoring-bot added a comment.Mar 19 2026, 9:56 PM

Cookbook cookbooks.sre.hosts.reimage was started by cdobbins@cumin2002 for host cp5024.eqsin.wmnet with OS trixie

ops-monitoring-bot added a comment.Mar 19 2026, 9:58 PM

Cookbook cookbooks.sre.hosts.reimage was started by cdobbins@cumin2002 for host cp5019.eqsin.wmnet with OS trixie

ops-monitoring-bot added a comment.Mar 19 2026, 11:18 PM

Cookbook cookbooks.sre.hosts.reimage started by cdobbins@cumin2002 for host cp5024.eqsin.wmnet with OS trixie executed with errors:

  • cp5024 (FAIL)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console cp5024.eqsin.wmnet" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.Mar 19 2026, 11:19 PM

Cookbook cookbooks.sre.hosts.reimage started by cdobbins@cumin2002 for host cp5019.eqsin.wmnet with OS trixie executed with errors:

  • cp5019 (FAIL)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console cp5019.eqsin.wmnet" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.Mar 19 2026, 11:59 PM

Cookbook cookbooks.sre.hosts.reimage was started by cdobbins@cumin2002 for host cp5019.eqsin.wmnet with OS trixie

ops-monitoring-bot added a comment.Mar 20 2026, 12:01 AM

Cookbook cookbooks.sre.hosts.reimage was started by cdobbins@cumin2002 for host cp5024.eqsin.wmnet with OS trixie

ops-monitoring-bot added a comment.Mar 20 2026, 12:01 AM

Cookbook cookbooks.sre.hosts.reimage started by cdobbins@cumin2002 for host cp5024.eqsin.wmnet with OS trixie executed with errors:

  • cp5024 (FAIL)
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console cp5024.eqsin.wmnet" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.Mar 20 2026, 1:21 AM

Cookbook cookbooks.sre.hosts.reimage started by cdobbins@cumin2002 for host cp5019.eqsin.wmnet with OS trixie completed:

  • cp5019 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh trixie OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202603200048_cdobbins_15578_cp5019.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
CDobbins updated the task description. (Show Details)Mar 20 2026, 5:31 AM
CDobbins updated the task description. (Show Details)Mar 20 2026, 7:15 PM
CDobbins updated the task description. (Show Details)Mar 20 2026, 9:57 PM
CDobbins updated the task description. (Show Details)Mar 23 2026, 8:34 PM
CDobbins updated the task description. (Show Details)Mar 23 2026, 8:43 PM
CDobbins updated the task description. (Show Details)Mar 23 2026, 9:03 PM
BCornwall updated the task description. (Show Details)Mar 24 2026, 1:42 AM
BCornwall updated the task description. (Show Details)Mar 30 2026, 4:11 PM
BCornwall updated the task description. (Show Details)Mar 30 2026, 5:57 PM
BCornwall updated the task description. (Show Details)Mar 30 2026, 7:47 PM
BCornwall updated the task description. (Show Details)Mar 30 2026, 8:51 PM
BCornwall updated the task description. (Show Details)Mar 30 2026, 9:17 PM
CDobbins updated the task description. (Show Details)Mar 30 2026, 10:48 PM
BCornwall updated the task description. (Show Details)Mar 30 2026, 11:03 PM
BCornwall updated the task description. (Show Details)Mar 31 2026, 5:52 PM
BCornwall updated the task description. (Show Details)Mar 31 2026, 7:26 PM
BCornwall updated the task description. (Show Details)Apr 1 2026, 9:23 PM
BCornwall updated the task description. (Show Details)
Vgutierrez closed subtask T417291: Upgrade the CDN to HAProxy 3.0 as Resolved.Apr 6 2026, 8:06 AM
BCornwall updated the task description. (Show Details)Apr 7 2026, 6:09 PM
ops-monitoring-bot added a comment.May 6 2026, 3:35 PM

Cookbook cookbooks.sre.hosts.reimage was started by sukhe@cumin1003 for host dns4004.wikimedia.org with OS bookworm

ops-monitoring-bot added a comment.May 6 2026, 4:37 PM

Cookbook cookbooks.sre.hosts.reimage started by sukhe@cumin1003 for host dns4004.wikimedia.org with OS bookworm completed:

  • dns4004 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Unable to disable Puppet, the host may have been unreachable
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202605061604_sukhe_773158_dns4004.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
ssingh added a comment.May 6 2026, 4:51 PM

^ The above is not a trixie reimage but a bookworm reimage for the ulsfo work in T424686. Please disregard.

Stashbot added a comment.May 19 2026, 6:07 PM

Mentioned in SAL (#wikimedia-operations) [2026-05-19T18:07:34Z] <cjd91> import gdnsd_3.99.0-alpha3~deb13u1 into trixie-wikimedia-T401832

BCornwall updated the task description. (Show Details)May 19 2026, 6:13 PM
ssingh mentioned this in T342154: Upgrade Traffic hosts to bookworm.May 22 2026, 3:33 PM
ops-monitoring-bot added a comment.Jun 17 2026, 2:28 PM

Cookbook cookbooks.sre.hosts.reimage was started by cdobbins@cumin1003 for host dns7002.wikimedia.org with OS trixie

gerritbot added a comment.Jun 17 2026, 4:03 PM

Change #1303481 had a related patch set uploaded (by CDobbins; author: CDobbins):

[operations/puppet@production] hieradata: override dns7002; use correct cfg file

https://gerrit.wikimedia.org/r/1303481

gerritbot added a project: Patch-For-Review.Jun 17 2026, 4:03 PM
gerritbot added a comment.Jun 17 2026, 4:06 PM

Change #1303481 merged by CDobbins:

[operations/puppet@production] hieradata: override dns7002; use correct cfg file

https://gerrit.wikimedia.org/r/1303481

gerritbot added a comment.Jun 17 2026, 5:21 PM

Change #1303497 had a related patch set uploaded (by CDobbins; author: CDobbins):

[operations/puppet@production] dnsrecursor: fix file name

https://gerrit.wikimedia.org/r/1303497

gerritbot added a comment.Jun 17 2026, 6:30 PM

Change #1303497 merged by CDobbins:

[operations/puppet@production] dnsrecursor: fix file name

https://gerrit.wikimedia.org/r/1303497

gerritbot added a comment.Jun 17 2026, 7:17 PM

Change #1303534 had a related patch set uploaded (by CDobbins; author: CDobbins):

[operations/puppet@production] dnsrecursor: add quotes to outgoing.dont_query vals

https://gerrit.wikimedia.org/r/1303534

gerritbot added a comment.Jun 17 2026, 7:34 PM

Change #1303534 merged by CDobbins:

[operations/puppet@production] dnsrecursor: add quotes to outgoing.dont_query vals

https://gerrit.wikimedia.org/r/1303534

gerritbot added a comment.Jun 17 2026, 7:41 PM

Change #1303540 had a related patch set uploaded (by CDobbins; author: CDobbins):

[operations/puppet@production] hieradata: override dns7002; use correct cfg file

https://gerrit.wikimedia.org/r/1303540

gerritbot added a comment.Jun 17 2026, 8:10 PM

Change #1303540 merged by CDobbins:

[operations/puppet@production] hieradata: override dns7002; use correct cfg file

https://gerrit.wikimedia.org/r/1303540

Maintenance_bot removed a project: Patch-For-Review.Jun 17 2026, 8:31 PM
ops-monitoring-bot added a comment.Jun 17 2026, 8:41 PM

Cookbook cookbooks.sre.hosts.reimage started by cdobbins@cumin1003 for host dns7002.wikimedia.org with OS trixie executed with errors:

  • dns7002 (FAIL)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh trixie OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202606171459_cdobbins_485339_dns7002.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console dns7002.wikimedia.org" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.Jun 18 2026, 5:37 PM

Cookbook cookbooks.sre.hosts.reimage was started by cdobbins@cumin2002 for host dns7002.wikimedia.org with OS bookworm

gerritbot added a comment.Jun 18 2026, 6:02 PM

Change #1304138 had a related patch set uploaded (by CDobbins; author: CDobbins):

[operations/puppet@production] hieradata: remove pdns cfg override for dns7002

https://gerrit.wikimedia.org/r/1304138

gerritbot added a project: Patch-For-Review.Jun 18 2026, 6:02 PM
gerritbot added a comment.Jun 18 2026, 6:09 PM

Change #1304138 merged by CDobbins:

[operations/puppet@production] hieradata: remove pdns cfg override for dns7002

https://gerrit.wikimedia.org/r/1304138

Maintenance_bot removed a project: Patch-For-Review.Jun 18 2026, 6:31 PM
ops-monitoring-bot added a comment.Jun 18 2026, 6:54 PM

Cookbook cookbooks.sre.hosts.reimage started by cdobbins@cumin2002 for host dns7002.wikimedia.org with OS bookworm completed:

  • dns7002 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202606181809_cdobbins_54040_dns7002.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
gerritbot added a comment.Jun 30 2026, 6:25 PM

Change #1306723 had a related patch set uploaded (by BCornwall; author: BCornwall):

[operations/puppet@production] varnish: Update tests container image to Trixie

https://gerrit.wikimedia.org/r/1306723

gerritbot added a project: Patch-For-Review.Jun 30 2026, 6:25 PM
gerritbot added a comment.Jun 30 2026, 6:31 PM

Change #1306723 merged by BCornwall:

[operations/puppet@production] varnish: Update tests container image to Trixie

https://gerrit.wikimedia.org/r/1306723

Maintenance_bot removed a project: Patch-For-Review.Jun 30 2026, 6:31 PM
ops-monitoring-bot added a comment.Jul 7 2026, 1:32 PM

Cookbook cookbooks.sre.hosts.reimage was started by cdobbins@cumin1003 for host dns7002.wikimedia.org with OS trixie

ops-monitoring-bot added a comment.Jul 7 2026, 3:33 PM

Cookbook cookbooks.sre.hosts.reimage started by cdobbins@cumin1003 for host dns7002.wikimedia.org with OS trixie executed with errors:

  • dns7002 (FAIL)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh trixie OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202607071400_cdobbins_2211547_dns7002.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console dns7002.wikimedia.org" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.Jul 7 2026, 3:33 PM

Cookbook cookbooks.sre.hosts.reimage was started by cdobbins@cumin1003 for host dns7002.wikimedia.org with OS bookworm

gerritbot added a comment.Jul 7 2026, 4:01 PM

Change #1308150 had a related patch set uploaded (by CDobbins; author: CDobbins):

[operations/puppet@production] hieradata: set `use_new_pdns_cfg` flag to true

https://gerrit.wikimedia.org/r/1308150

gerritbot added a project: Patch-For-Review.Jul 7 2026, 4:01 PM
ops-monitoring-bot added a comment.Jul 7 2026, 4:58 PM

Cookbook cookbooks.sre.hosts.reimage started by cdobbins@cumin1003 for host dns7002.wikimedia.org with OS bookworm completed:

  • dns7002 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202607071606_cdobbins_2251860_dns7002.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
gerritbot added a comment.Mon, Jul 13, 1:55 PM

Change #1308150 merged by CDobbins:

[operations/puppet@production] hieradata: issue fixes

https://gerrit.wikimedia.org/r/1308150

ops-monitoring-bot added a comment.Mon, Jul 13, 2:05 PM

Cookbook cookbooks.sre.hosts.reimage was started by cdobbins@cumin2002 for host dns7002.wikimedia.org with OS trixie

Maintenance_bot removed a project: Patch-For-Review.Mon, Jul 13, 2:32 PM
ops-monitoring-bot added a comment.Mon, Jul 13, 6:19 PM

Cookbook cookbooks.sre.hosts.reimage started by cdobbins@cumin2002 for host dns7002.wikimedia.org with OS trixie completed:

  • dns7002 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh trixie OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202607131435_cdobbins_2038041_dns7002.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Wed, Jul 15, 5:47 PM

Cookbook cookbooks.sre.hosts.reimage was started by cdobbins@cumin2003 for host dns7002.wikimedia.org with OS trixie

ops-monitoring-bot added a comment.Wed, Jul 15, 6:59 PM

Cookbook cookbooks.sre.hosts.reimage started by cdobbins@cumin2003 for host dns7002.wikimedia.org with OS trixie completed:

  • dns7002 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh trixie OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202607151819_cdobbins_1769559_dns7002.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits