Page MenuHomePhabricator
Create Task
Maniphest T342154

Upgrade Traffic hosts to bookworm
Closed, ResolvedPublic
Actions

Description

This task tracks the upgrade of the Traffic hosts to bookworm, affecting the services below and identified by their cumin aliases.

This is meant to be an umbrella task for all changes that will be part of this upgrade, such as the Debian packaging, Puppet changes, and the related testing include reimaging.

Progress:

HostDebian PackagesReimaging
cpnot done, upgraded to trixie in T401832not done, upgraded to trixie in T401832
acmechiefdonedone
ncredirdonedone
dnsdonedone
durumdonedone
wikidoughdonedone
lvsN/A (no Python 2)N/A
lvs experimental
pybal-testN/A (no Python 2)N/A

Debian Packaging

This includes packages that we build ourselves.

A:cp

A:acmechief

  • acme-chief

A:dns (rec/auth)

  • gdnsd (gdnsd_3.99.0~alpha2-2_amd64.changes)
  • pdns-recursor (already in Debian proper)

A:durum

A:wikidough

  • anycast-healthchecker (anycast-healthchecker_0.9.1-1+wmf12u1_amd64.changes)
  • dnsdist (dnsdist_1.8.0-1+wmf12u1_amd64.changes)

A:lvs

N/A

lvs experimental

A:pybaltest

N/A

Bookworm reimaging

A:acmechief

  • acmechief2001.codfw.wmnet
  • acmechief2002.codfw.wmnet
  • acmechief1001.eqiad.wmnet
  • acmechief-test1001.eqiad.wmnet
  • acmechief-test2001.codfw.wmnet

A:cp

  • cp1075.eqiad.wmnet
  • cp1076.eqiad.wmnet
  • cp1077.eqiad.wmnet
  • cp1078.eqiad.wmnet
  • cp1079.eqiad.wmnet
  • cp1080.eqiad.wmnet
  • cp1081.eqiad.wmnet
  • cp1082.eqiad.wmnet
  • cp1083.eqiad.wmnet
  • cp1084.eqiad.wmnet
  • cp1085.eqiad.wmnet
  • cp1086.eqiad.wmnet
  • cp1087.eqiad.wmnet
  • cp1088.eqiad.wmnet
  • cp1089.eqiad.wmnet
  • cp1090.eqiad.wmnet
  • cp2027.codfw.wmnet
  • cp2028.codfw.wmnet
  • cp2029.codfw.wmnet
  • cp2030.codfw.wmnet
  • cp2031.codfw.wmnet
  • cp2032.codfw.wmnet
  • cp2033.codfw.wmnet
  • cp2034.codfw.wmnet
  • cp2035.codfw.wmnet
  • cp2036.codfw.wmnet
  • cp2037.codfw.wmnet
  • cp2038.codfw.wmnet
  • cp2039.codfw.wmnet
  • cp2040.codfw.wmnet
  • cp2041.codfw.wmnet
  • cp2042.codfw.wmnet
  • cp3066.esams.wmnet
  • cp3067.esams.wmnet
  • cp3068.esams.wmnet
  • cp3069.esams.wmnet
  • cp3070.esams.wmnet
  • cp3071.esams.wmnet
  • cp3072.esams.wmnet
  • cp3073.esams.wmnet
  • cp3074.esams.wmnet
  • cp3075.esams.wmnet
  • cp3076.esams.wmnet
  • cp3077.esams.wmnet
  • cp3078.esams.wmnet
  • cp3079.esams.wmnet
  • cp3080.esams.wmnet
  • cp3081.esams.wmnet
  • cp4037.ulsfo.wmnet
  • cp4038.ulsfo.wmnet
  • cp4039.ulsfo.wmnet
  • cp4040.ulsfo.wmnet
  • cp4041.ulsfo.wmnet
  • cp4042.ulsfo.wmnet
  • cp4043.ulsfo.wmnet
  • cp4044.ulsfo.wmnet
  • cp4045.ulsfo.wmnet
  • cp4046.ulsfo.wmnet
  • cp4047.ulsfo.wmnet
  • cp4048.ulsfo.wmnet
  • cp4049.ulsfo.wmnet
  • cp4050.ulsfo.wmnet
  • cp4051.ulsfo.wmnet
  • cp4052.ulsfo.wmnet
  • cp5017.eqsin.wmnet
  • cp5018.eqsin.wmnet
  • cp5019.eqsin.wmnet
  • cp5020.eqsin.wmnet
  • cp5021.eqsin.wmnet
  • cp5022.eqsin.wmnet
  • cp5023.eqsin.wmnet
  • cp5024.eqsin.wmnet
  • cp5025.eqsin.wmnet
  • cp5026.eqsin.wmnet
  • cp5027.eqsin.wmnet
  • cp5028.eqsin.wmnet
  • cp5029.eqsin.wmnet
  • cp5030.eqsin.wmnet
  • cp5031.eqsin.wmnet
  • cp5032.eqsin.wmnet
  • cp6001.drmrs.wmnet
  • cp6002.drmrs.wmnet
  • cp6003.drmrs.wmnet
  • cp6004.drmrs.wmnet
  • cp6005.drmrs.wmnet
  • cp6006.drmrs.wmnet
  • cp6007.drmrs.wmnet
  • cp6008.drmrs.wmnet
  • cp6009.drmrs.wmnet
  • cp6010.drmrs.wmnet
  • cp6011.drmrs.wmnet
  • cp6012.drmrs.wmnet
  • cp6013.drmrs.wmnet
  • cp6014.drmrs.wmnet
  • cp6015.drmrs.wmnet
  • cp6016.drmrs.wmnet

A:durum

  • durum1001.eqiad.wmnet
  • durum1002.eqiad.wmnet
  • durum2001.codfw.wmnet
  • durum2002.codfw.wmnet
  • durum4001.ulsfo.wmnet
  • durum4002.ulsfo.wmnet
  • durum5001.eqsin.wmnet
  • durum5002.eqsin.wmnet
  • durum6001.drmrs.wmnet
  • durum6002.drmrs.wmnet

A:ncredir

  • ncredir1001.eqiad.wmnet
  • ncredir1002.eqiad.wmnet
  • ncredir2001.codfw.wmnet
  • ncredir2002.codfw.wmnet
  • ncredir3003.esams.wmnet
  • ncredir3004.esams.wmnet
  • ncredir4001.ulsfo.wmnet
  • ncredir4002.ulsfo.wmnet
  • ncredir5001.eqsin.wmnet
  • ncredir5002.eqsin.wmnet
  • ncredir6001.drmrs.wmnet
  • ncredir6002.drmrs.wmnet

A:dns-rec (and auth)

  • dns1004.wikimedia.org
  • dns1005.wikimedia.org
  • dns1006.wikimedia.org
  • dns2004.wikimedia.org
  • dns2005.wikimedia.org
  • dns2006.wikimedia.org
  • dns3003.wikimedia.org
  • dns3004.wikimedia.org
  • dns4003.wikimedia.org
  • dns4004.wikimedia.org
  • dns5003.wikimedia.org
  • dns5004.wikimedia.org
  • dns6001.wikimedia.org
  • dns6002.wikimedia.org

A:wikidough

  • doh1001.wikimedia.org
  • doh1002.wikimedia.org
  • doh2001.wikimedia.org
  • doh2002.wikimedia.org
  • doh3003.wikimedia.org
  • doh3004.wikimedia.org
  • doh4001.wikimedia.org
  • doh4002.wikimedia.org
  • doh5001.wikimedia.org
  • doh5002.wikimedia.org
  • doh6001.wikimedia.org
  • doh6002.wikimedia.org

Details

Related Changes in Gerrit:
SubjectAuthorRepoBranchLines +/-
Version 6.0.11-1wm2 for Debian BookwormFabfuroperations/debs/varnish4debian-wmf+10 -2
acme-chief: Remove acmechief2002 passive hostBCornwalloperations/puppetproduction+0 -1
acme-chief: Remove acmechief2001 passive hostBCornwalloperations/puppetproduction+0 -1
acme_chief: Remove acmechief1001 passive hostBCornwalloperations/puppetproduction+0 -1
acme-chief: Switch acmechief_host to acmechief1001BCornwalloperations/puppetproduction+1 -1
acme-chief: Set acmechief1001 as activeBCornwalloperations/puppetproduction+2 -2
acme_chief: Set acmechief-test1001 as active hostBCornwalloperations/puppetproduction+4 -4
hiera: remove dns3004 from authdns_serversBCornwalloperations/puppetproduction+0 -1
hiera: remove dns3003 from authdns_serversBCornwalloperations/puppetproduction+0 -1
hiera: remove dns2006 from authdns_serversBCornwalloperations/puppetproduction+0 -1
hiera: remove dns2005 from authdns_serversBCornwalloperations/puppetproduction+0 -1
hiera: remove dns2004 from authdns_serversBCornwalloperations/puppetproduction+0 -1
hiera: remove dns1006 from authdns_serversBCornwalloperations/puppetproduction+0 -1
hiera: remove dns1005 from authdns_serversBCornwalloperations/puppetproduction+0 -1
hiera: remove dns1004 for authdns_servers for reimagingSsinghoperations/puppetproduction+0 -1
hiera: remove dns5004 from authdns_serversBCornwalloperations/puppetproduction+0 -1
hiera: remove dns5003 from authdns_serversBCornwalloperations/puppetproduction+0 -1
hiera: remove dns4004 from authdns_serversBCornwalloperations/puppetproduction+0 -1
hiera: remove dns6002 from authdns_serversBCornwalloperations/puppetproduction+0 -1
hiera: remove dns6001 from authdns_serversBCornwalloperations/puppetproduction+0 -1
hiera: remove dns4003 for authdns_servers for reimagingSsinghoperations/puppetproduction+0 -1
Update dependencies to match Bookworm versionsBCornwalloperations/software/acme-chiefmaster+21 -23
Release 0.36-2 for BookwormBCornwalloperations/software/acme-chiefdebian+31 -22
Release 9.1.4-1wm2 to target BookwormBCornwalloperations/debs/trafficservermaster+9 -2
aptrepo: Add Bookworm HAProxy third party reposBCornwalloperations/puppetproduction+20 -7
OpenSSL 3 compat for update-ocsp scriptBBlackoperations/puppetproduction+19 -4
varnish:common: add bookworm version for PythonSsinghoperations/puppetproduction+2 -1
Release 9.2.1-1wm2Ssinghoperations/debs/trafficservermaster+8 -1
Release 1.5.3-5Ssinghoperations/software/varnish/libvmod-re2debian-6.0+33 -0
Release 1.9-4 to target BookwormBCornwalloperations/software/varnish/libvmod-netmapperdebian+10 -2
Rebuild against BullseyeBCornwalloperations/debs/varnish-modulesmaster+8 -1
Release 3.99.0~alpha2-2Ssinghoperations/debs/gdnsdmaster+7 -50
Release 0.9.1-1+wmf12u1Ssinghoperations/debs/python-anycast-healthcheckermaster+34 -11
Release 0.5 for bookwormBCornwalloperations/software/varnish/libvmod-querysortmain+9 -1
Release 1.1.0-3Fabfuroperations/software/varnish/varnishkafkadebian+11 -113
Version 0.4.6-4Fabfuroperations/debs/python-logstashmaster+11 -0
Release 0.20Fabfuroperations/software/purgedmaster+14 -19
Release 0.1-3Fabfuroperations/debs/prometheus-varnishkafka-exportermaster+11 -1
fifo-log-demux: Add socat as companion packageFabfuroperations/puppetproduction+3 -0
Release 0.4Fabfuroperations/software/prometheus-rdkafka-exportermaster+8 -0
aptrepo: add component/dnsdist to bookwormSsinghoperations/puppetproduction+1 -0
Release 2.0.0-4Fabfuroperations/debs/file-read-backwardsdebian+12 -1
Release 0.6.4Fabfuroperations/software/fifo-log-demuxmaster+15 -3
Release dnsdist 1.8.0-1+wmf12u1Ssinghoperations/debs/dnsdistmaster+6 -0
Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
gerritbot added a comment.Oct 23 2023, 7:26 PM

Change 967976 had a related patch set uploaded (by BCornwall; author: BCornwall):

[operations/puppet@production] hiera: remove dns4004 from authdns_servers

https://gerrit.wikimedia.org/r/967976

gerritbot added a comment.Oct 23 2023, 8:12 PM

Change 967976 merged by BCornwall:

[operations/puppet@production] hiera: remove dns4004 from authdns_servers

https://gerrit.wikimedia.org/r/967976

ops-monitoring-bot added a comment.Oct 23 2023, 8:18 PM

Cookbook cookbooks.sre.hosts.reimage was started by brett@cumin2002 for host dns4004.wikimedia.org with OS bookworm

ops-monitoring-bot added a comment.Oct 23 2023, 9:37 PM

Cookbook cookbooks.sre.hosts.reimage started by brett@cumin2002 for host dns4004.wikimedia.org with OS bookworm completed:

  • dns4004 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202310232044_brett_3618436_dns4004.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
BCornwall updated the task description. (Show Details)Oct 23 2023, 10:06 PM
gerritbot added a comment.Oct 24 2023, 4:46 PM

Change 968294 had a related patch set uploaded (by BCornwall; author: BCornwall):

[operations/puppet@production] hiera: remove dns5003 from authdns_servers

https://gerrit.wikimedia.org/r/968294

gerritbot added a comment.Oct 24 2023, 4:47 PM

Change 968294 merged by BCornwall:

[operations/puppet@production] hiera: remove dns5003 from authdns_servers

https://gerrit.wikimedia.org/r/968294

ops-monitoring-bot added a comment.Oct 24 2023, 4:59 PM

Cookbook cookbooks.sre.hosts.reimage was started by brett@cumin2002 for host dns5003.wikimedia.org with OS bookworm

ops-monitoring-bot added a comment.Oct 24 2023, 6:54 PM

Cookbook cookbooks.sre.hosts.reimage started by brett@cumin2002 for host dns5003.wikimedia.org with OS bookworm completed:

  • dns5003 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202310241749_brett_3845170_dns5003.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
gerritbot added a comment.Oct 24 2023, 7:11 PM

Change 968342 had a related patch set uploaded (by BCornwall; author: BCornwall):

[operations/puppet@production] hiera: remove dns5004 from authdns_servers

https://gerrit.wikimedia.org/r/968342

gerritbot added a comment.Oct 24 2023, 7:14 PM

Change 968342 merged by BCornwall:

[operations/puppet@production] hiera: remove dns5004 from authdns_servers

https://gerrit.wikimedia.org/r/968342

ops-monitoring-bot added a comment.Oct 24 2023, 7:23 PM

Cookbook cookbooks.sre.hosts.reimage was started by brett@cumin2002 for host dns5004.wikimedia.org with OS bookworm

ops-monitoring-bot added a comment.Oct 24 2023, 9:08 PM

Cookbook cookbooks.sre.hosts.reimage started by brett@cumin2002 for host dns5004.wikimedia.org with OS bookworm completed:

  • dns5004 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202310242014_brett_3876277_dns5004.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
BCornwall updated the task description. (Show Details)Oct 24 2023, 9:16 PM
gerritbot added a comment.Oct 25 2023, 2:01 PM

Change 968680 had a related patch set uploaded (by Ssingh; author: Ssingh):

[operations/puppet@production] hiera: remove dns1004 for authdns_servers for reimaging

https://gerrit.wikimedia.org/r/968680

gerritbot added a comment.Oct 25 2023, 2:03 PM

Change 968680 merged by Ssingh:

[operations/puppet@production] hiera: remove dns1004 for authdns_servers for reimaging

https://gerrit.wikimedia.org/r/968680

ssingh updated the task description. (Show Details)Oct 25 2023, 3:14 PM
gerritbot added a comment.Oct 25 2023, 4:30 PM

Change 968721 had a related patch set uploaded (by BCornwall; author: BCornwall):

[operations/puppet@production] hiera: remove dns1005 from authdns_servers

https://gerrit.wikimedia.org/r/968721

gerritbot added a comment.Oct 25 2023, 4:36 PM

Change 968721 merged by BCornwall:

[operations/puppet@production] hiera: remove dns1005 from authdns_servers

https://gerrit.wikimedia.org/r/968721

ops-monitoring-bot added a comment.Oct 25 2023, 4:45 PM

Cookbook cookbooks.sre.hosts.reimage was started by brett@cumin2002 for host dns1005.wikimedia.org with OS bookworm

ops-monitoring-bot added a comment.Oct 25 2023, 5:28 PM

Cookbook cookbooks.sre.hosts.reimage started by brett@cumin2002 for host dns1005.wikimedia.org with OS bookworm completed:

  • dns1005 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202310251702_brett_4115897_dns1005.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
gerritbot added a comment.Oct 25 2023, 6:16 PM

Change 968735 had a related patch set uploaded (by BCornwall; author: BCornwall):

[operations/puppet@production] hiera: remove dns1006 from authdns_servers

https://gerrit.wikimedia.org/r/968735

gerritbot added a comment.Oct 25 2023, 6:35 PM

Change 968735 merged by BCornwall:

[operations/puppet@production] hiera: remove dns1006 from authdns_servers

https://gerrit.wikimedia.org/r/968735

ops-monitoring-bot added a comment.Oct 25 2023, 7:27 PM

Cookbook cookbooks.sre.hosts.reimage was started by brett@cumin2002 for host dns1006.wikimedia.org with OS bookworm

ops-monitoring-bot added a comment.Oct 25 2023, 8:04 PM

Cookbook cookbooks.sre.hosts.reimage started by brett@cumin2002 for host dns1006.wikimedia.org with OS bookworm completed:

  • dns1006 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202310251944_brett_4150404_dns1006.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
BCornwall updated the task description. (Show Details)Oct 25 2023, 8:26 PM
gerritbot added a comment.Oct 26 2023, 6:32 PM

Change 969185 had a related patch set uploaded (by BCornwall; author: BCornwall):

[operations/puppet@production] hiera: remove dns2001 from authdns_servers

https://gerrit.wikimedia.org/r/969185

gerritbot added a comment.Oct 26 2023, 6:33 PM

Change 969185 merged by BCornwall:

[operations/puppet@production] hiera: remove dns2004 from authdns_servers

https://gerrit.wikimedia.org/r/969185

ops-monitoring-bot added a comment.Oct 26 2023, 6:44 PM

Cookbook cookbooks.sre.hosts.reimage was started by brett@cumin2002 for host dns2004.wikimedia.org with OS bookworm

ops-monitoring-bot added a comment.Oct 26 2023, 7:38 PM

Cookbook cookbooks.sre.hosts.reimage started by brett@cumin2002 for host dns2004.wikimedia.org with OS bookworm completed:

  • dns2004 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202310261908_brett_216869_dns2004.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
gerritbot added a comment.Oct 26 2023, 7:54 PM

Change 969194 had a related patch set uploaded (by BCornwall; author: BCornwall):

[operations/puppet@production] hiera: remove dns2005 from authdns_servers

https://gerrit.wikimedia.org/r/969194

gerritbot added a comment.Oct 26 2023, 7:56 PM

Change 969194 merged by BCornwall:

[operations/puppet@production] hiera: remove dns2005 from authdns_servers

https://gerrit.wikimedia.org/r/969194

ops-monitoring-bot added a comment.Oct 26 2023, 8:03 PM

Cookbook cookbooks.sre.hosts.reimage was started by brett@cumin2002 for host dns2005.wikimedia.org with OS bookworm

ops-monitoring-bot added a comment.Oct 26 2023, 8:41 PM

Cookbook cookbooks.sre.hosts.reimage started by brett@cumin2002 for host dns2005.wikimedia.org with OS bookworm completed:

  • dns2005 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202310262023_brett_237506_dns2005.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
gerritbot added a comment.Oct 26 2023, 9:39 PM

Change 969212 had a related patch set uploaded (by BCornwall; author: BCornwall):

[operations/puppet@production] hiera: remove dns2006 from authdns_servers

https://gerrit.wikimedia.org/r/969212

gerritbot added a comment.Oct 26 2023, 9:40 PM

Change 969212 merged by BCornwall:

[operations/puppet@production] hiera: remove dns2006 from authdns_servers

https://gerrit.wikimedia.org/r/969212

ops-monitoring-bot added a comment.Oct 26 2023, 9:47 PM

Cookbook cookbooks.sre.hosts.reimage was started by brett@cumin2002 for host dns2006.wikimedia.org with OS bookworm

ops-monitoring-bot added a comment.Oct 26 2023, 10:49 PM

Cookbook cookbooks.sre.hosts.reimage started by brett@cumin2002 for host dns2006.wikimedia.org with OS bookworm completed:

  • dns2006 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202310262210_brett_260534_dns2006.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
BCornwall updated the task description. (Show Details)Oct 26 2023, 11:01 PM
gerritbot added a comment.Oct 30 2023, 5:05 PM

Change 969931 had a related patch set uploaded (by BCornwall; author: BCornwall):

[operations/puppet@production] hiera: remove dns3003 from authdns_servers

https://gerrit.wikimedia.org/r/969931

gerritbot added a comment.Oct 30 2023, 5:06 PM

Change 969931 merged by BCornwall:

[operations/puppet@production] hiera: remove dns3003 from authdns_servers

https://gerrit.wikimedia.org/r/969931

ops-monitoring-bot added a comment.Oct 30 2023, 5:12 PM

Cookbook cookbooks.sre.hosts.reimage was started by brett@cumin2002 for host dns3003.wikimedia.org with OS bookworm

ops-monitoring-bot added a comment.Oct 30 2023, 6:38 PM

Cookbook cookbooks.sre.hosts.reimage started by brett@cumin2002 for host dns3003.wikimedia.org with OS bookworm completed:

  • dns3003 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202310301742_brett_1204482_dns3003.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
gerritbot added a comment.Oct 30 2023, 6:51 PM

Change 969949 had a related patch set uploaded (by BCornwall; author: BCornwall):

[operations/puppet@production] hiera: remove dns3004 from authdns_servers

https://gerrit.wikimedia.org/r/969949

gerritbot added a comment.Oct 30 2023, 6:51 PM

Change 969949 merged by BCornwall:

[operations/puppet@production] hiera: remove dns3004 from authdns_servers

https://gerrit.wikimedia.org/r/969949

ops-monitoring-bot added a comment.Oct 30 2023, 7:21 PM

Cookbook cookbooks.sre.hosts.reimage was started by brett@cumin2002 for host dns3004.wikimedia.org with OS bookworm

ops-monitoring-bot added a comment.Oct 30 2023, 8:20 PM

Cookbook cookbooks.sre.hosts.reimage started by brett@cumin2002 for host dns3004.wikimedia.org with OS bookworm completed:

  • dns3004 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202310301951_brett_1232033_dns3004.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
BCornwall updated the task description. (Show Details)Oct 30 2023, 8:50 PM
BCornwall updated the task description. (Show Details)Oct 31 2023, 4:41 PM
CodeReviewBot added a comment.Oct 31 2023, 6:38 PM

brett merged https://gitlab.wikimedia.org/repos/sre/acme-chief/-/merge_requests/3

Update dependencies to match Bookworm versions

BCornwall changed the status of subtask T344330: acme-chief should support debian bookworm from Open to In Progress.Oct 31 2023, 10:28 PM
CodeReviewBot added a comment.Nov 2 2023, 9:11 PM

brett merged https://gitlab.wikimedia.org/repos/sre/acme-chief/-/merge_requests/2

Release 0.36-2 for Bookworm

ops-monitoring-bot added a comment.Nov 8 2023, 7:49 PM

Cookbook cookbooks.sre.hosts.reimage was started by brett@cumin2002 for host acmechief-test1001.eqiad.wmnet with OS bookworm

BCornwall updated the task description. (Show Details)Nov 8 2023, 7:52 PM
gerritbot added a comment.Nov 8 2023, 7:56 PM

Change 972886 had a related patch set uploaded (by BCornwall; author: BCornwall):

[operations/puppet@production] acme_chief: Set acmechief-test1001 as active host

https://gerrit.wikimedia.org/r/972886

ops-monitoring-bot added a comment.Nov 8 2023, 8:28 PM

Cookbook cookbooks.sre.hosts.reimage started by brett@cumin2002 for host acmechief-test1001.eqiad.wmnet with OS bookworm completed:

  • acmechief-test1001 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via gnt-instance
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Set boot media to disk
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202311082005_brett_1618392_acmechief-test1001.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
BCornwall updated the task description. (Show Details)Nov 8 2023, 8:37 PM
BCornwall removed a project: SRE.Nov 9 2023, 5:28 PM
gerritbot added a comment.Nov 9 2023, 6:04 PM

Change 972886 merged by BCornwall:

[operations/puppet@production] acme_chief: Set acmechief-test1001 as active host

https://gerrit.wikimedia.org/r/972886

ops-monitoring-bot added a comment.Nov 9 2023, 6:31 PM

Cookbook cookbooks.sre.hosts.reimage was started by brett@cumin2002 for host acmechief-test2001.codfw.wmnet with OS bookworm

ops-monitoring-bot added a comment.Nov 9 2023, 7:04 PM

Cookbook cookbooks.sre.hosts.reimage started by brett@cumin2002 for host acmechief-test2001.codfw.wmnet with OS bookworm completed:

  • acmechief-test2001 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via gnt-instance
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Set boot media to disk
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202311091851_brett_2257575_acmechief-test2001.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ssingh updated the task description. (Show Details)Nov 13 2023, 3:20 PM
ssingh updated the task description. (Show Details)
BCornwall closed subtask T344330: acme-chief should support debian bookworm as Resolved.Nov 13 2023, 6:17 PM
Stashbot added a comment.Nov 13 2023, 6:42 PM

Mentioned in SAL (#wikimedia-operations) [2023-11-13T18:42:55Z] <sukhe> pool cp4052 as first cp host for bookworm testing: T342154

BCornwall updated the task description. (Show Details)Nov 14 2023, 4:16 PM
ops-monitoring-bot added a comment.Nov 16 2023, 4:17 PM

Cookbook cookbooks.sre.hosts.reimage was started by brett@cumin1001 for host acmechief1001.eqiad.wmnet with OS bookworm

gerritbot added a comment.Nov 16 2023, 4:46 PM

Change 975024 had a related patch set uploaded (by BCornwall; author: BCornwall):

[operations/puppet@production] acme_chief: Remove acmechief1001 passive host

https://gerrit.wikimedia.org/r/975024

ops-monitoring-bot added a comment.Nov 16 2023, 4:49 PM

Cookbook cookbooks.sre.hosts.reimage started by brett@cumin1001 for host acmechief1001.eqiad.wmnet with OS bookworm completed:

  • acmechief1001 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via gnt-instance
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Set boot media to disk
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202311161631_brett_1915187_acmechief1001.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
Stashbot added a comment.Nov 16 2023, 5:00 PM

Mentioned in SAL (#wikimedia-operations) [2023-11-16T17:00:22Z] <brett> Disabling puppet on all acme-chief clients for acme-chief bookworm upgrades - T342154

gerritbot added a comment.Nov 16 2023, 5:04 PM

Change 975046 had a related patch set uploaded (by BCornwall; author: BCornwall):

[operations/puppet@production] acme-chief: Set acmechief1001 as active

https://gerrit.wikimedia.org/r/975046

gerritbot added a comment.Nov 16 2023, 5:08 PM

Change 975046 merged by BCornwall:

[operations/puppet@production] acme-chief: Set acmechief1001 as active

https://gerrit.wikimedia.org/r/975046

gerritbot added a comment.Nov 16 2023, 5:14 PM

Change 975047 had a related patch set uploaded (by BCornwall; author: BCornwall):

[operations/puppet@production] acme-chief: Switch acmechief_host to acmechief1001

https://gerrit.wikimedia.org/r/975047

gerritbot added a comment.Nov 16 2023, 5:19 PM

Change 975047 merged by BCornwall:

[operations/puppet@production] acme-chief: Switch acmechief_host to acmechief1001

https://gerrit.wikimedia.org/r/975047

Stashbot added a comment.Nov 16 2023, 5:27 PM

Mentioned in SAL (#wikimedia-operations) [2023-11-16T17:26:59Z] <brett> Re-enabling puppet on all acme-chief clients post-bookworm upgrade - T342154

gerritbot added a comment.Nov 16 2023, 5:47 PM

Change 975024 abandoned by BCornwall:

[operations/puppet@production] acme_chief: Remove acmechief1001 passive host

Reason:

No need any more

https://gerrit.wikimedia.org/r/975024

BCornwall updated the task description. (Show Details)Nov 16 2023, 6:33 PM
gerritbot added a comment.Nov 20 2023, 4:18 PM

Change 975853 had a related patch set uploaded (by BCornwall; author: BCornwall):

[operations/puppet@production] acme-chief: Remove acmechief2001 passive host

https://gerrit.wikimedia.org/r/975853

gerritbot added a comment.Nov 20 2023, 6:18 PM

Change 975853 merged by BCornwall:

[operations/puppet@production] acme-chief: Remove acmechief2001 passive host

https://gerrit.wikimedia.org/r/975853

ops-monitoring-bot added a comment.Nov 20 2023, 6:27 PM

Cookbook cookbooks.sre.hosts.reimage was started by brett@cumin1001 for host acmechief2001.codfw.wmnet with OS bookworm

ops-monitoring-bot added a comment.Nov 20 2023, 7:03 PM

Cookbook cookbooks.sre.hosts.reimage started by brett@cumin1001 for host acmechief2001.codfw.wmnet with OS bookworm completed:

  • acmechief2001 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via gnt-instance
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Set boot media to disk
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202311201844_brett_255399_acmechief2001.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
gerritbot added a comment.Nov 20 2023, 8:04 PM

Change 975911 had a related patch set uploaded (by BCornwall; author: BCornwall):

[operations/puppet@production] acme-chief: Remove acmechief2002 passive host

https://gerrit.wikimedia.org/r/975911

BCornwall updated the task description. (Show Details)Nov 21 2023, 4:02 PM
BCornwall updated the task description. (Show Details)
gerritbot added a comment.Nov 21 2023, 5:04 PM

Change 975911 abandoned by BCornwall:

[operations/puppet@production] acme-chief: Remove acmechief2002 passive host

Reason:

Already upgraded

https://gerrit.wikimedia.org/r/975911

Vgutierrez updated the task description. (Show Details)Dec 5 2023, 10:07 AM
KOfori updated the task description. (Show Details)Dec 8 2023, 4:29 PM
BCornwall changed the task status from In Progress to Stalled.Apr 9 2024, 8:47 PM

Stalling as cp hosts cannot be updated until the performance issues with openssl 3.x are dealt with (T352744)

BCornwall moved this task from Actively Servicing to Backlog on the Traffic board.Jul 23 2024, 6:28 PM
hashar mentioned this in T371001: Migrate docker-registry.wikimedia.org/releng/operations-dnslint from Buster to Bookworm.Jul 25 2024, 10:03 AM
Vgutierrez closed subtask T339134: Package and deploy ATS 9.2.5 as Resolved.Sep 5 2024, 1:05 PM
CDanis subscribed.Jul 29 2025, 6:47 PM
ssingh added a comment.Sep 29 2025, 2:06 PM

Because of the regressions observed in OpenSSL 3.x, we never finished the upgrade for the cp hosts to bookworm. This task is stalled while we go through the possible upgrade paths for either OpenSSL, or an upgrade to trixie itself, depending on the former.

ssingh added a comment.May 22 2026, 3:33 PM

The only blocker in this task was the cp hosts for OpenSSL. We have already upgraded them to trixie in T401832, so this task can be resolved. A note has been made for the same in the description.

ssingh closed this task as Resolved.May 22 2026, 3:34 PM
ssingh claimed this task.
ssingh updated the task description. (Show Details)
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits