Let's upgrade es4 to Bullseye.
es4 is RW not RO, so it does require a proper DB switchover.
- es2022
- es2021
- es2020
- es1022
- es1021 (master)
- es1020
Description
Description
Details
Details
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Open | None | T291916 Tracking task for Bullseye migrations in production | |||
| Resolved | Marostegui | T298585 Upgrade WMF database-and-backup-related hosts to bullseye | |||
| Resolved | Marostegui | T300005 Upgrade es4 to Bullseye | |||
| Resolved | Marostegui | T300127 Switchover es4 master es1021 -> es1020 |
Event Timeline
Comment Actions
Change 756952 had a related patch set uploaded (by Marostegui; author: Marostegui):
[operations/puppet@production] es2020: Disable notifications
Comment Actions
Change 756952 merged by Marostegui:
[operations/puppet@production] es2020: Disable notifications
Comment Actions
Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1001 for host es2020.codfw.wmnet with OS bullseye
Comment Actions
Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1001 for host es2020.codfw.wmnet with OS bullseye completed:
- es2020 (PASS)
- Downtimed on Icinga
- Disabled Puppet
- Removed from Puppet and PuppetDB if present
- Deleted any existing Puppet certificate
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Host up (new fresh bullseye OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202201251002_marostegui_9575_es2020.out
- Checked BIOS boot parameters are back to normal
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- Updated Netbox data from PuppetDB
Comment Actions
Change 756961 had a related patch set uploaded (by Marostegui; author: Marostegui):
[operations/puppet@production] es2021: Disable notifications
Comment Actions
Change 756961 merged by Marostegui:
[operations/puppet@production] es2021: Disable notifications
Comment Actions
Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1001 for host es2021.codfw.wmnet with OS bullseye
Comment Actions
Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1001 for host es2021.codfw.wmnet with OS bullseye completed:
- es2021 (WARN)
- Downtimed on Icinga
- Disabled Puppet
- Removed from Puppet and PuppetDB if present
- Deleted any existing Puppet certificate
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Host up (new fresh bullseye OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202201251055_marostegui_28151_es2021.out
- Checked BIOS boot parameters are back to normal
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is not optimal, downtime not removed
- Updated Netbox data from PuppetDB
Comment Actions
Mentioned in SAL (#wikimedia-operations) [2022-01-26T06:31:50Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Depool es1020 T300005', diff saved to https://phabricator.wikimedia.org/P19231 and previous config saved to /var/cache/conftool/dbconfig/20220126-063149-marostegui.json
Comment Actions
Change 757287 had a related patch set uploaded (by Marostegui; author: Marostegui):
[operations/puppet@production] es1020: Disable notifications
Comment Actions
Change 757287 merged by Marostegui:
[operations/puppet@production] es1020: Disable notifications
Comment Actions
Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1001 for host es1020.eqiad.wmnet with OS bullseye
Comment Actions
Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1001 for host es1020.eqiad.wmnet with OS bullseye executed with errors:
- es1020 (FAIL)
- Downtimed on Icinga
- Disabled Puppet
- Removed from Puppet and PuppetDB if present
- Deleted any existing Puppet certificate
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- The reimage failed, see the cookbook logs for the details
Comment Actions
Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1001 for host es1020.eqiad.wmnet with OS bullseye
Comment Actions
Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1001 for host es1020.eqiad.wmnet with OS bullseye completed:
- es1020 (WARN)
- Removed from Puppet and PuppetDB if present
- Deleted any existing Puppet certificate
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Host up (new fresh bullseye OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202201261015_marostegui_25345_es1020.out
- Checked BIOS boot parameters are back to normal
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is not optimal, downtime not removed
- Updated Netbox data from PuppetDB
Comment Actions
On Wednesday around 9AM UTC I will disable writes to es4 so I can do a switchover and then reimage the current master.
Comment Actions
Change 759207 had a related patch set uploaded (by Marostegui; author: Marostegui):
[operations/puppet@production] es1021: Disable notifications
Comment Actions
Change 759207 merged by Marostegui:
[operations/puppet@production] es1021: Disable notifications
Comment Actions
Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1001 for host es1021.eqiad.wmnet with OS bullseye
Comment Actions
Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1001 for host es1021.eqiad.wmnet with OS bullseye completed:
- es1021 (WARN)
- Downtimed on Icinga
- Disabled Puppet
- Removed from Puppet and PuppetDB if present
- Deleted any existing Puppet certificate
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Host up (new fresh bullseye OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202202020939_marostegui_14176_es1021.out
- Checked BIOS boot parameters are back to normal
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is not optimal, downtime not removed
- Updated Netbox data from PuppetDB
Comment Actions
This is all done, es1021 is being repooled slowly.
(6) es[2020-2022].codfw.wmnet,es[1020-1022].eqiad.wmnet ----- OUTPUT of 'lsb_release -a' ----- Distributor ID: Debian Description: Debian GNU/Linux 11 (bullseye) Release: 11 Codename: bullseye No LSB modules are available.