Page MenuHomePhabricator
Create Task
Maniphest T362134

Upgrade s3 to MariaDB 10.6
Closed, ResolvedPublic
Actions

Description

Bookworm + MariaDB 10.6

  • dbstore1007
  • db2209
  • db2205
  • db2194
  • db2190
  • db2186
  • db2177
  • db2156
  • db2149
  • db2139 (backup source T360751)
  • db2127 master
  • db2109
  • db2105
  • db1240 (backup source T360751)
  • db1223 candidate master
  • db1212
  • db1198
  • db1189
  • db1175
  • db1166
  • db1157 master
  • db1154
  • db1150 (backup source T360751)
  • clouddb1021
  • clouddb1017
  • clouddb1013

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
db1223: Disable notificationsoperations/puppetproduction+1 -0
db1212: Disable notificationsoperations/puppetproduction+1 -0
Customize query in gerrit

Related Objects
Search...

Event Timeline

Marostegui created this task.Apr 9 2024, 5:45 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 9 2024, 5:45 AM
Marostegui triaged this task as Medium priority.Apr 9 2024, 5:45 AM
Marostegui moved this task from Triage to In progress on the DBA board.
Marostegui mentioned this in T356960: Upgrade hosts to MariaDB 10.6.
Stashbot added a comment.Apr 10 2024, 4:55 AM

Mentioned in SAL (#wikimedia-operations) [2024-04-10T04:55:36Z] <marostegui@cumin1002> dbctl commit (dc=all): 'Depool db1223 T362134', diff saved to https://phabricator.wikimedia.org/P60175 and previous config saved to /var/cache/conftool/dbconfig/20240410-045534-marostegui.json

Stashbot added a comment.Apr 10 2024, 4:57 AM

Mentioned in SAL (#wikimedia-operations) [2024-04-10T04:57:10Z] <marostegui@cumin1002> dbctl commit (dc=all): 'Depool db1166 T362134', diff saved to https://phabricator.wikimedia.org/P60177 and previous config saved to /var/cache/conftool/dbconfig/20240410-045710-marostegui.json

ops-monitoring-bot added a comment.Apr 10 2024, 4:58 AM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1002 for host db1166.eqiad.wmnet with OS bookworm

Marostegui updated the task description. (Show Details)Apr 10 2024, 5:25 AM
ops-monitoring-bot added a comment.Apr 10 2024, 5:33 AM

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1002 for host db1166.eqiad.wmnet with OS bookworm completed:

  • db1166 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202404100512_marostegui_1731347_db1166.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Apr 10 2024, 10:19 AM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1002 for host db1175.eqiad.wmnet with OS bookworm

Marostegui updated the task description. (Show Details)Apr 10 2024, 10:48 AM
ops-monitoring-bot added a comment.Apr 10 2024, 10:56 AM

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1002 for host db1175.eqiad.wmnet with OS bookworm completed:

  • db1175 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202404101035_marostegui_1776392_db1175.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Apr 11 2024, 5:15 AM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1002 for host db1189.eqiad.wmnet with OS bookworm

Marostegui updated the task description. (Show Details)Apr 11 2024, 5:44 AM
ops-monitoring-bot added a comment.Apr 11 2024, 5:52 AM

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1002 for host db1189.eqiad.wmnet with OS bookworm completed:

  • db1189 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202404110531_marostegui_1938192_db1189.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Apr 11 2024, 8:06 AM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1002 for host db1198.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Apr 11 2024, 8:36 AM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1002 for host db1198.eqiad.wmnet with OS bookworm

Marostegui updated the task description. (Show Details)Apr 11 2024, 8:51 AM
ops-monitoring-bot added a comment.Apr 11 2024, 8:55 AM

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1002 for host db1198.eqiad.wmnet with OS bookworm completed:

  • db1198 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202404110840_marostegui_1981766_db1198.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
    • Cleared switch DHCP cache and MAC table for the host IP and MAC (EVPN Switch)
ops-monitoring-bot added a comment.Apr 11 2024, 11:10 AM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1002 for host db2177.codfw.wmnet with OS bookworm

ops-monitoring-bot added a comment.Apr 11 2024, 11:52 AM

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1002 for host db2177.codfw.wmnet with OS bookworm completed:

  • db2177 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202404111131_marostegui_2022530_db2177.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
    • Cleared switch DHCP cache and MAC table for the host IP and MAC (EVPN Switch)
Marostegui updated the task description. (Show Details)Apr 11 2024, 12:06 PM
ops-monitoring-bot added a comment.Apr 11 2024, 1:50 PM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1002 for host db2149.codfw.wmnet with OS bookworm

Marostegui updated the task description. (Show Details)Apr 11 2024, 2:22 PM
ops-monitoring-bot added a comment.Apr 11 2024, 2:30 PM

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1002 for host db2149.codfw.wmnet with OS bookworm completed:

  • db2149 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202404111410_marostegui_2053992_db2149.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Apr 12 2024, 5:17 AM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1002 for host db2109.codfw.wmnet with OS bookworm

Marostegui mentioned this in T360751: Upgrade backup sources to MariaDB 10.6.Apr 12 2024, 5:41 AM
Marostegui updated the task description. (Show Details)
Marostegui updated the task description. (Show Details)
Marostegui updated the task description. (Show Details)Apr 12 2024, 5:53 AM
ops-monitoring-bot added a comment.Apr 12 2024, 5:56 AM

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1002 for host db2109.codfw.wmnet with OS bookworm completed:

  • db2109 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202404120535_marostegui_2178039_db2109.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
    • Cleared switch DHCP cache and MAC table for the host IP and MAC (EVPN Switch)
ops-monitoring-bot added a comment.Apr 16 2024, 5:04 AM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1002 for host db2156.codfw.wmnet with OS bookworm

Marostegui updated the task description. (Show Details)Apr 16 2024, 5:25 AM
Marostegui updated the task description. (Show Details)Apr 16 2024, 5:41 AM
ops-monitoring-bot added a comment.Apr 16 2024, 5:49 AM

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1002 for host db2156.codfw.wmnet with OS bookworm completed:

  • db2156 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202404160526_marostegui_2888069_db2156.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
    • Cleared switch DHCP cache and MAC table for the host IP and MAC (EVPN Switch)
ops-monitoring-bot added a comment.Apr 16 2024, 7:52 AM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1002 for host db2105.codfw.wmnet with OS bookworm

ops-monitoring-bot added a comment.Apr 16 2024, 8:34 AM

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1002 for host db2105.codfw.wmnet with OS bookworm completed:

  • db2105 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202404160813_marostegui_2909726_db2105.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
    • Cleared switch DHCP cache and MAC table for the host IP and MAC (EVPN Switch)
Marostegui updated the task description. (Show Details)Apr 16 2024, 8:48 AM
Marostegui added a subtask: T362616: Switchover s3 master (db2127 -> db2205).
Marostegui closed subtask T362616: Switchover s3 master (db2127 -> db2205) as Resolved.Apr 16 2024, 3:11 PM
ops-monitoring-bot added a comment.Apr 16 2024, 3:13 PM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1002 for host db2127.codfw.wmnet with OS bookworm

Marostegui updated the task description. (Show Details)Apr 16 2024, 3:49 PM
ops-monitoring-bot added a comment.Apr 16 2024, 3:57 PM

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1002 for host db2127.codfw.wmnet with OS bookworm completed:

  • db2127 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202404161534_marostegui_2979348_db2127.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
gerritbot added a comment.Apr 29 2024, 6:05 AM

Change #1025002 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] db1212: Disable notifications

https://gerrit.wikimedia.org/r/1025002

gerritbot added a project: Patch-For-Review.Apr 29 2024, 6:05 AM

Change #1025002 merged by Marostegui:

[operations/puppet@production] db1212: Disable notifications

https://gerrit.wikimedia.org/r/1025002

ops-monitoring-bot added a comment.Apr 29 2024, 6:06 AM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1002 for host db1212.eqiad.wmnet with OS bookworm

Maintenance_bot removed a project: Patch-For-Review.Apr 29 2024, 6:30 AM
Marostegui updated the task description. (Show Details)Apr 29 2024, 6:37 AM
ops-monitoring-bot added a comment.Apr 29 2024, 6:46 AM

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1002 for host db1212.eqiad.wmnet with OS bookworm completed:

  • db1212 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202404290623_marostegui_2976934_db1212.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
gerritbot added a comment.Apr 29 2024, 8:08 AM

Change #1025275 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] db1223: Disable notifications

https://gerrit.wikimedia.org/r/1025275

gerritbot added a project: Patch-For-Review.Apr 29 2024, 8:08 AM

Change #1025275 merged by Marostegui:

[operations/puppet@production] db1223: Disable notifications

https://gerrit.wikimedia.org/r/1025275

ops-monitoring-bot added a comment.Apr 29 2024, 8:09 AM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1002 for host db1223.eqiad.wmnet with OS bookworm

Maintenance_bot removed a project: Patch-For-Review.Apr 29 2024, 8:30 AM
Marostegui updated the task description. (Show Details)Apr 29 2024, 8:37 AM
ops-monitoring-bot added a comment.Apr 29 2024, 8:45 AM

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1002 for host db1223.eqiad.wmnet with OS bookworm completed:

  • db1223 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202404290824_marostegui_2997477_db1223.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
Marostegui added a subtask: T363672: Switchover s3 master (db1157 -> db1223).Apr 29 2024, 8:48 AM
ops-monitoring-bot added a comment.Apr 30 2024, 5:19 AM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1002 for host db1157.eqiad.wmnet with OS bookworm

Marostegui closed subtask T363672: Switchover s3 master (db1157 -> db1223) as Resolved.Apr 30 2024, 5:20 AM
Marostegui updated the task description. (Show Details)Apr 30 2024, 5:48 AM
ops-monitoring-bot added a comment.Apr 30 2024, 5:55 AM

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1002 for host db1157.eqiad.wmnet with OS bookworm completed:

  • db1157 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202404300535_marostegui_3190679_db1157.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
Marostegui closed this task as Resolved.Apr 30 2024, 5:56 AM
Marostegui added a subscriber: jcrespo.

@jcrespo s3 is fully done.

Maintenance_bot moved this task from In progress to Done on the DBA board.Apr 30 2024, 6:29 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits