Let's make s6 Debian Trixie entirely
Description
Description
Details
Details
Related Changes in Gerrit:
Customize query in gerrit
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Open | Marostegui | T422365 Migration to Debian Trixie of production database-related hosts | |||
| Resolved | Marostegui | T422777 Migrate s6 section to Debian Trixie | |||
| Resolved | Marostegui | T423837 Switchover s6 master (db2229 -> db2214) | |||
| Resolved | Marostegui | T424522 Switchover s6 master (db1173 -> db1201) |
Event Timeline
There are a very large number of changes, so older changes are hidden. Show Older Changes
Comment Actions
I will wait as usual to be the last of the replicas to be upgraded. One question, as a reimage could cause a package upgrade- please let me know which is the latest version that can be installed, as dbprov hosts should have the same or higher minor version for the backups to be considered safe.
Other than those, I think reasonable considerations, the backup sources can be upgraded at any time, ideally during the UTC day to not disturb ongoing backups. Only s4 sources are continuously queried for commons media backups, and that requires more coordination (pausing media backups when done).
Let me know if this works for you @Marostegui.
Comment Actions
That works for me @jcrespo - thank you.
Also keep in mind those hosts aren't blocking us in anyway as the version is the same.
The repos are up-to-date with safe packages, everything there is stable.
Comment Actions
Change #1270451 had a related patch set uploaded (by Marostegui; author: Marostegui):
[operations/puppet@production] db2224: Disable notifications
Comment Actions
Change #1270451 merged by Marostegui:
[operations/puppet@production] db2224: Disable notifications
Comment Actions
Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1003 for host db2224.codfw.wmnet with OS trixie
Comment Actions
Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1003 for host db2224.codfw.wmnet with OS trixie completed:
- db2224 (WARN)
- Downtimed on Icinga/Alertmanager
- Disabled Puppet
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Checked BIOS boot parameters are back to normal
- Host up (new fresh trixie OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- Removed previous downtime on Alertmanager (old OS)
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202604131440_marostegui_2456049_db2224.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is not optimal, downtime not removed
- Updated Netbox data from PuppetDB
Comment Actions
Change #1270478 had a related patch set uploaded (by Marostegui; author: Marostegui):
[operations/puppet@production] db1187.yaml: Disable notifications
Comment Actions
Change #1270478 merged by Marostegui:
[operations/puppet@production] db1187.yaml: Disable notifications
Comment Actions
Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1003 for host db1187.eqiad.wmnet with OS trixie
Comment Actions
Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1003 for host db1187.eqiad.wmnet with OS trixie completed:
- db1187 (WARN)
- Downtimed on Icinga/Alertmanager
- Disabled Puppet
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Checked BIOS boot parameters are back to normal
- Host up (new fresh trixie OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- Removed previous downtime on Alertmanager (old OS)
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202604131529_marostegui_2509042_db1187.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is not optimal, downtime not removed
- Updated Netbox data from PuppetDB
Comment Actions
Change #1270760 had a related patch set uploaded (by Marostegui; author: Marostegui):
[operations/puppet@production] db2217: Disable notifications
Comment Actions
Change #1270760 merged by Marostegui:
[operations/puppet@production] db2217: Disable notifications
Comment Actions
Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1003 for host db2217.codfw.wmnet with OS trixie
Comment Actions
Change #1270761 had a related patch set uploaded (by Marostegui; author: Marostegui):
[operations/puppet@production] db1180: Disable notifications
Comment Actions
Change #1270761 merged by Marostegui:
[operations/puppet@production] db1180: Disable notifications
Comment Actions
Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1003 for host db1180.eqiad.wmnet with OS trixie
Comment Actions
Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1003 for host db2217.codfw.wmnet with OS trixie completed:
- db2217 (PASS)
- Downtimed on Icinga/Alertmanager
- Disabled Puppet
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Checked BIOS boot parameters are back to normal
- Host up (new fresh trixie OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- Removed previous downtime on Alertmanager (old OS)
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202604140554_marostegui_2965533_db2217.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- Updated Netbox data from PuppetDB
Comment Actions
Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1003 for host db1180.eqiad.wmnet with OS trixie completed:
- db1180 (WARN)
- Downtimed on Icinga/Alertmanager
- Disabled Puppet
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Checked BIOS boot parameters are back to normal
- Host up (new fresh trixie OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- Removed previous downtime on Alertmanager (old OS)
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202604140608_marostegui_2966264_db1180.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is not optimal, downtime not removed
- Updated Netbox data from PuppetDB
Comment Actions
Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1003 for host db2180.codfw.wmnet with OS trixie
Comment Actions
$ export HTTP_PROXY=http://proxy_user_id:proxy_password@proxy_server:proxy_port
$ export HTTPS_PROXY=http://proxy_user_id:proxy_password@proxy_server:proxy_port
Comment Actions
Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1003 for host db2180.codfw.wmnet with OS trixie completed:
- db2180 (PASS)
- Downtimed on Icinga/Alertmanager
- Disabled Puppet
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Checked BIOS boot parameters are back to normal
- Host up (new fresh trixie OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- Removed previous downtime on Alertmanager (old OS)
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202604140738_marostegui_2979531_db2180.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- Updated Netbox data from PuppetDB
Comment Actions
Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1003 for host db1168.eqiad.wmnet with OS trixie
Comment Actions
Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1003 for host db1168.eqiad.wmnet with OS trixie completed:
- db1168 (WARN)
- Downtimed on Icinga/Alertmanager
- Disabled Puppet
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Checked BIOS boot parameters are back to normal
- Host up (new fresh trixie OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- Removed previous downtime on Alertmanager (old OS)
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202604140831_marostegui_3035630_db1168.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is not optimal, downtime not removed
- Updated Netbox data from PuppetDB
Comment Actions
Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1003 for host db2214.codfw.wmnet with OS trixie
Comment Actions
Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1003 for host db2214.codfw.wmnet with OS trixie completed:
- db2214 (WARN)
- Downtimed on Icinga/Alertmanager
- Disabled Puppet
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Checked BIOS boot parameters are back to normal
- Host up (new fresh trixie OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- Removed previous downtime on Alertmanager (old OS)
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202604141331_marostegui_176658_db2214.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is not optimal, downtime not removed
- Updated Netbox data from PuppetDB
Comment Actions
Change #1272372 had a related patch set uploaded (by Marostegui; author: Marostegui):
[operations/puppet@production] db2193: Disable notifications
Comment Actions
Change #1272372 merged by Marostegui:
[operations/puppet@production] db2193: Disable notifications
Comment Actions
Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1003 for host db2193.codfw.wmnet with OS trixie
Comment Actions
Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1003 for host db2193.codfw.wmnet with OS trixie completed:
- db2193 (WARN)
- Downtimed on Icinga/Alertmanager
- Disabled Puppet
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Checked BIOS boot parameters are back to normal
- Host up (new fresh trixie OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- Removed previous downtime on Alertmanager (old OS)
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202604160614_marostegui_1320255_db2193.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is not optimal, downtime not removed
- Updated Netbox data from PuppetDB
Comment Actions
Change #1272455 had a related patch set uploaded (by Marostegui; author: Marostegui):
[operations/puppet@production] db2169: Disable notifications
Comment Actions
Change #1272455 merged by Marostegui:
[operations/puppet@production] db2169: Disable notifications
Comment Actions
Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1003 for host db2169.codfw.wmnet with OS trixie
Comment Actions
Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1003 for host db2169.codfw.wmnet with OS trixie completed:
- db2169 (WARN)
- Downtimed on Icinga/Alertmanager
- Disabled Puppet
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Checked BIOS boot parameters are back to normal
- Host up (new fresh trixie OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- Removed previous downtime on Alertmanager (old OS)
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202604160805_marostegui_1349177_db2169.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is not optimal, downtime not removed
- Updated Netbox data from PuppetDB
Comment Actions
Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1003 for host db2158.codfw.wmnet with OS trixie
Comment Actions
Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1003 for host db2158.codfw.wmnet with OS trixie completed:
- db2158 (WARN)
- Downtimed on Icinga/Alertmanager
- Disabled Puppet
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Checked BIOS boot parameters are back to normal
- Host up (new fresh trixie OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- Removed previous downtime on Alertmanager (old OS)
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202604170539_marostegui_1851327_db2158.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is not optimal, downtime not removed
- Updated Netbox data from PuppetDB
Comment Actions
Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1003 for host db1201.eqiad.wmnet with OS trixie
Comment Actions
Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1003 for host db1201.eqiad.wmnet with OS trixie completed:
- db1201 (WARN)
- Downtimed on Icinga/Alertmanager
- Disabled Puppet
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Checked BIOS boot parameters are back to normal
- Host up (new fresh trixie OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- Removed previous downtime on Alertmanager (old OS)
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202604170624_marostegui_1859338_db1201.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is not optimal, downtime not removed
- Updated Netbox data from PuppetDB
Comment Actions
Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1003 for host db2151.codfw.wmnet with OS trixie
Comment Actions
Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1003 for host db2151.codfw.wmnet with OS trixie completed:
- db2151 (WARN)
- Downtimed on Icinga/Alertmanager
- Disabled Puppet
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Checked BIOS boot parameters are back to normal
- Host up (new fresh trixie OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- Removed previous downtime on Alertmanager (old OS)
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202604200559_marostegui_2921124_db2151.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is not optimal, downtime not removed
- Updated Netbox data from PuppetDB
Comment Actions
Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1003 for host db2229.codfw.wmnet with OS trixie
Comment Actions
Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1003 for host db2229.codfw.wmnet with OS trixie completed:
- db2229 (WARN)
- Downtimed on Icinga/Alertmanager
- Disabled Puppet
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Checked BIOS boot parameters are back to normal
- Host up (new fresh trixie OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- Removed previous downtime on Alertmanager (old OS)
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202604200714_marostegui_2932434_db2229.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is not optimal, downtime not removed
- Updated Netbox data from PuppetDB
Comment Actions
Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1003 for host db1165.eqiad.wmnet with OS trixie
Comment Actions
Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1003 for host db1165.eqiad.wmnet with OS trixie completed:
- db1165 (WARN)
- Downtimed on Icinga/Alertmanager
- Disabled Puppet
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Checked BIOS boot parameters are back to normal
- Host up (new fresh trixie OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- Removed previous downtime on Alertmanager (old OS)
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202604200849_marostegui_3020048_db1165.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is not optimal, downtime not removed
- Updated Netbox data from PuppetDB
Comment Actions
Change #1278038 had a related patch set uploaded (by Marostegui; author: Marostegui):
[operations/puppet@production] db1173: Disable notifications
Comment Actions
Change #1278038 merged by Marostegui:
[operations/puppet@production] db1173: Disable notifications
Comment Actions
All done except the non production hosts, which will be tracked in their own task attached to their hostnames.