Page MenuHomePhabricator
Create Task
Maniphest T354506

Upgrade s6 hosts to Bookworm
Closed, ResolvedPublic
Actions

Description

Upgrade s6 core hosts to Debian Bookworm and MariaDB 10.6
Please note this only includes core hosts:

  • master: db1173
  • db1213
  • db1187
  • db1168
  • db1231
  • db1201
  • db1165
  • db1180
  • db1224
  • master: db2114
  • db2193
  • db2124
  • db2158
  • db2169
  • db2180
  • db2129
  • db2171
  • db2117
  • db2151

Details

Related Changes in Gerrit:
SubjectAuthorRepoBranchLines +/-
db2114: Disable notificationsMarosteguioperations/puppetproduction+2 -1
db2129: Disable notificationsMarosteguioperations/puppetproduction+1 -0
db1173 Disable notificationsMarosteguioperations/puppetproduction+2 -1
db2171: Disable notificationsMarosteguioperations/puppetproduction+1 -0
db1231: Disable notificationsMarosteguioperations/puppetproduction+1 -0
db1165: Disable notificationsMarosteguioperations/puppetproduction+1 -0
db2158: Disable notificationsMarosteguioperations/puppetproduction+1 -0
db1213: Disable notificationsMarosteguioperations/puppetproduction+1 -0
db2169: Disable notificationsMarosteguioperations/puppetproduction+1 -0
db1187: Disable notificationsMarosteguioperations/puppetproduction+1 -0
db1224: Remove package declarationMarosteguioperations/puppetproduction+0 -1
db2124: Remove package declarationMarosteguioperations/puppetproduction+0 -1
db2117: Remove package declarationMarosteguioperations/puppetproduction+0 -1
db1168: Disable notificationsMarosteguioperations/puppetproduction+1 -0
db2124: Disable notificationsMarosteguioperations/puppetproduction+1 -0
db1201: Disable notificationsMarosteguioperations/puppetproduction+1 -0
db2180: Remove package declarationMarosteguioperations/puppetproduction+0 -1
db2180: Disable notificationsMarosteguioperations/puppetproduction+1 -0
db1180: Disable notificationsMarosteguioperations/puppetproduction+1 -0
db2151: Disable notificationsMarosteguioperations/puppetproduction+1 -0
db1224: Disable notificationsMarosteguioperations/puppetproduction+1 -0
db2117: Disable notificationsMarosteguioperations/puppetproduction+1 -0
Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
gerritbot added a project: Patch-For-Review.Jan 19 2024, 8:10 AM

Change 991700 merged by Marostegui:

[operations/puppet@production] db1224: Remove package declaration

https://gerrit.wikimedia.org/r/991700

Maintenance_bot removed a project: Patch-For-Review.Jan 19 2024, 8:30 AM
gerritbot added a comment.Jan 22 2024, 6:07 AM

Change 991983 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] db1187: Disable notifications

https://gerrit.wikimedia.org/r/991983

gerritbot added a project: Patch-For-Review.Jan 22 2024, 6:08 AM
Stashbot added a comment.Jan 22 2024, 6:08 AM

Mentioned in SAL (#wikimedia-operations) [2024-01-22T06:08:12Z] <marostegui@cumin1002> dbctl commit (dc=all): 'Depool db1187 T354506', diff saved to https://phabricator.wikimedia.org/P55121 and previous config saved to /var/cache/conftool/dbconfig/20240122-060811-marostegui.json

gerritbot added a comment.Jan 22 2024, 6:09 AM

Change 991983 merged by Marostegui:

[operations/puppet@production] db1187: Disable notifications

https://gerrit.wikimedia.org/r/991983

ops-monitoring-bot added a comment.Jan 22 2024, 6:10 AM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1002 for host db1187.eqiad.wmnet with OS bookworm

Maintenance_bot removed a project: Patch-For-Review.Jan 22 2024, 6:30 AM
Marostegui updated the task description. (Show Details)Jan 22 2024, 6:40 AM
ops-monitoring-bot added a comment.Jan 22 2024, 6:46 AM

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1002 for host db1187.eqiad.wmnet with OS bookworm completed:

  • db1187 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202401220626_marostegui_3437658_db1187.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
gerritbot added a comment.Jan 22 2024, 6:50 AM

Change 991984 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] db2169: Disable notifications

https://gerrit.wikimedia.org/r/991984

gerritbot added a project: Patch-For-Review.Jan 22 2024, 6:50 AM

Change 991984 merged by Marostegui:

[operations/puppet@production] db2169: Disable notifications

https://gerrit.wikimedia.org/r/991984

ops-monitoring-bot added a comment.Jan 22 2024, 6:52 AM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1002 for host db2169.codfw.wmnet with OS bookworm

Maintenance_bot removed a project: Patch-For-Review.Jan 22 2024, 7:31 AM
ops-monitoring-bot added a comment.Jan 22 2024, 7:36 AM

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1002 for host db2169.codfw.wmnet with OS bookworm completed:

  • db2169 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202401220714_marostegui_3445967_db2169.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
Marostegui updated the task description. (Show Details)Jan 22 2024, 7:46 AM
gerritbot added a comment.Jan 22 2024, 8:39 AM

Change 992080 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] db1213: Disable notifications

https://gerrit.wikimedia.org/r/992080

gerritbot added a project: Patch-For-Review.Jan 22 2024, 8:39 AM
ops-monitoring-bot added a comment.Jan 22 2024, 8:40 AM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1002 for host db1213.eqiad.wmnet with OS bookworm

gerritbot added a comment.Jan 22 2024, 8:40 AM

Change 992080 merged by Marostegui:

[operations/puppet@production] db1213: Disable notifications

https://gerrit.wikimedia.org/r/992080

Marostegui updated the task description. (Show Details)Jan 22 2024, 9:10 AM
ops-monitoring-bot added a comment.Jan 22 2024, 9:17 AM

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1002 for host db1213.eqiad.wmnet with OS bookworm completed:

  • db1213 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202401220856_marostegui_3466930_db1213.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
Maintenance_bot removed a project: Patch-For-Review.Jan 22 2024, 9:31 AM
gerritbot added a comment.Jan 22 2024, 10:16 AM

Change 992092 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] db2158: Disable notifications

https://gerrit.wikimedia.org/r/992092

gerritbot added a project: Patch-For-Review.Jan 22 2024, 10:16 AM

Change 992092 merged by Marostegui:

[operations/puppet@production] db2158: Disable notifications

https://gerrit.wikimedia.org/r/992092

ops-monitoring-bot added a comment.Jan 22 2024, 10:18 AM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1002 for host db2158.codfw.wmnet with OS bookworm

Maintenance_bot removed a project: Patch-For-Review.Jan 22 2024, 10:31 AM
ops-monitoring-bot added a comment.Jan 22 2024, 11:01 AM

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1002 for host db2158.codfw.wmnet with OS bookworm completed:

  • db2158 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202401221040_marostegui_3482799_db2158.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
Marostegui updated the task description. (Show Details)Jan 22 2024, 11:07 AM
gerritbot added a comment.Jan 22 2024, 1:20 PM

Change 992160 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] db1165: Disable notifications

https://gerrit.wikimedia.org/r/992160

gerritbot added a project: Patch-For-Review.Jan 22 2024, 1:20 PM

Change 992160 merged by Marostegui:

[operations/puppet@production] db1165: Disable notifications

https://gerrit.wikimedia.org/r/992160

ops-monitoring-bot added a comment.Jan 22 2024, 1:21 PM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1002 for host db1165.eqiad.wmnet with OS bookworm

Stashbot added a comment.Jan 22 2024, 1:22 PM

Mentioned in SAL (#wikimedia-operations) [2024-01-22T13:22:01Z] <marostegui> Upgrade sanitarium master, there will be lag on s6 wiki replicas T354506

Maintenance_bot removed a project: Patch-For-Review.Jan 22 2024, 1:30 PM
ops-monitoring-bot added a comment.Jan 22 2024, 1:57 PM

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1002 for host db1165.eqiad.wmnet with OS bookworm completed:

  • db1165 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202401221336_marostegui_3513772_db1165.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
Marostegui updated the task description. (Show Details)Jan 22 2024, 2:18 PM
gerritbot added a comment.Jan 23 2024, 6:55 AM

Change 992353 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] db1231: Disable notifications

https://gerrit.wikimedia.org/r/992353

gerritbot added a project: Patch-For-Review.Jan 23 2024, 6:55 AM

Change 992353 merged by Marostegui:

[operations/puppet@production] db1231: Disable notifications

https://gerrit.wikimedia.org/r/992353

ops-monitoring-bot added a comment.Jan 23 2024, 6:57 AM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1002 for host db1231.eqiad.wmnet with OS bookworm

Marostegui updated the task description. (Show Details)Jan 23 2024, 7:30 AM
Maintenance_bot removed a project: Patch-For-Review.Jan 23 2024, 7:30 AM
ops-monitoring-bot added a comment.Jan 23 2024, 7:34 AM

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1002 for host db1231.eqiad.wmnet with OS bookworm completed:

  • db1231 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202401230713_marostegui_3671818_db1231.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
gerritbot added a comment.Jan 23 2024, 10:10 AM

Change 992389 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] db2171: Disable notifications

https://gerrit.wikimedia.org/r/992389

gerritbot added a project: Patch-For-Review.Jan 23 2024, 10:10 AM

Change 992389 merged by Marostegui:

[operations/puppet@production] db2171: Disable notifications

https://gerrit.wikimedia.org/r/992389

ops-monitoring-bot added a comment.Jan 23 2024, 10:12 AM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1002 for host db2171.codfw.wmnet with OS bookworm

Marostegui updated the task description. (Show Details)Jan 23 2024, 10:53 AM
ops-monitoring-bot added a comment.Jan 23 2024, 10:55 AM

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1002 for host db2171.codfw.wmnet with OS bookworm completed:

  • db2171 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202401231035_marostegui_3710747_db2171.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
Marostegui added a subtask: T355660: Switchover s6 master (db1173 -> db1231).Jan 23 2024, 10:59 AM
Maintenance_bot removed a project: Patch-For-Review.Jan 23 2024, 11:30 AM
Marostegui closed subtask T355660: Switchover s6 master (db1173 -> db1231) as Resolved.Jan 23 2024, 11:39 AM
gerritbot added a comment.Jan 23 2024, 2:11 PM

Change 992420 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] db1173 Disable notifications

https://gerrit.wikimedia.org/r/992420

gerritbot added a project: Patch-For-Review.Jan 23 2024, 2:11 PM
ops-monitoring-bot added a comment.Jan 23 2024, 2:12 PM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1002 for host db1173.eqiad.wmnet with OS bookworm

gerritbot added a comment.Jan 23 2024, 2:12 PM

Change 992420 merged by Marostegui:

[operations/puppet@production] db1173 Disable notifications

https://gerrit.wikimedia.org/r/992420

Maintenance_bot removed a project: Patch-For-Review.Jan 23 2024, 2:31 PM
Marostegui updated the task description. (Show Details)Jan 23 2024, 2:39 PM
ops-monitoring-bot added a comment.Jan 23 2024, 2:47 PM

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1002 for host db1173.eqiad.wmnet with OS bookworm completed:

  • db1173 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202401231426_marostegui_3748101_db1173.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
Stashbot added a comment.Jan 24 2024, 5:56 AM

Mentioned in SAL (#wikimedia-operations) [2024-01-24T05:56:36Z] <marostegui@cumin1002> dbctl commit (dc=all): 'Depool db2129 T354506', diff saved to https://phabricator.wikimedia.org/P55454 and previous config saved to /var/cache/conftool/dbconfig/20240124-055635-marostegui.json

gerritbot added a comment.Jan 24 2024, 5:57 AM

Change 992556 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] db2129: Disable notifications

https://gerrit.wikimedia.org/r/992556

gerritbot added a project: Patch-For-Review.Jan 24 2024, 5:57 AM
ops-monitoring-bot added a comment.Jan 24 2024, 5:58 AM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1002 for host db2129.codfw.wmnet with OS bookworm

gerritbot added a comment.Jan 24 2024, 5:58 AM

Change 992556 merged by Marostegui:

[operations/puppet@production] db2129: Disable notifications

https://gerrit.wikimedia.org/r/992556

Maintenance_bot removed a project: Patch-For-Review.Jan 24 2024, 6:30 AM
Marostegui updated the task description. (Show Details)Jan 24 2024, 6:32 AM
ops-monitoring-bot added a comment.Jan 24 2024, 6:40 AM

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1002 for host db2129.codfw.wmnet with OS bookworm completed:

  • db2129 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202401240618_marostegui_3883383_db2129.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
Stashbot added a comment.Jan 31 2024, 6:19 AM

Mentioned in SAL (#wikimedia-operations) [2024-01-31T06:19:32Z] <marostegui@cumin1002> dbctl commit (dc=all): 'Depool db2114 T354506', diff saved to https://phabricator.wikimedia.org/P55908 and previous config saved to /var/cache/conftool/dbconfig/20240131-061932-root.json

gerritbot added a comment.Jan 31 2024, 6:20 AM

Change 994610 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] db2114: Disable notifications

https://gerrit.wikimedia.org/r/994610

gerritbot added a project: Patch-For-Review.Jan 31 2024, 6:20 AM
ops-monitoring-bot added a comment.Jan 31 2024, 6:22 AM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1002 for host db2114.codfw.wmnet with OS bookworm

gerritbot added a comment.Jan 31 2024, 6:25 AM

Change 994610 merged by Marostegui:

[operations/puppet@production] db2114: Disable notifications

https://gerrit.wikimedia.org/r/994610

ops-monitoring-bot added a comment.Jan 31 2024, 6:54 AM

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1002 for host db2114.codfw.wmnet with OS bookworm completed:

  • db2114 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202401310639_marostegui_1149424_db2114.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
Stashbot added a comment.Jan 31 2024, 7:01 AM

Mentioned in SAL (#wikimedia-operations) [2024-01-31T07:01:11Z] <marostegui@cumin1002> dbctl commit (dc=all): 'db2114 (re)pooling @ 1%: After Bookworm upgrade T354506', diff saved to https://phabricator.wikimedia.org/P55916 and previous config saved to /var/cache/conftool/dbconfig/20240131-070111-root.json

Marostegui updated the task description. (Show Details)Jan 31 2024, 7:06 AM
Stashbot added a comment.Jan 31 2024, 7:16 AM

Mentioned in SAL (#wikimedia-operations) [2024-01-31T07:16:16Z] <marostegui@cumin1002> dbctl commit (dc=all): 'db2114 (re)pooling @ 5%: After Bookworm upgrade T354506', diff saved to https://phabricator.wikimedia.org/P55919 and previous config saved to /var/cache/conftool/dbconfig/20240131-071616-root.json

Stashbot added a comment.Jan 31 2024, 7:31 AM

Mentioned in SAL (#wikimedia-operations) [2024-01-31T07:31:22Z] <marostegui@cumin1002> dbctl commit (dc=all): 'db2114 (re)pooling @ 10%: After Bookworm upgrade T354506', diff saved to https://phabricator.wikimedia.org/P55922 and previous config saved to /var/cache/conftool/dbconfig/20240131-073121-root.json

Stashbot added a comment.Jan 31 2024, 7:46 AM

Mentioned in SAL (#wikimedia-operations) [2024-01-31T07:46:27Z] <marostegui@cumin1002> dbctl commit (dc=all): 'db2114 (re)pooling @ 25%: After Bookworm upgrade T354506', diff saved to https://phabricator.wikimedia.org/P55924 and previous config saved to /var/cache/conftool/dbconfig/20240131-074627-root.json

Stashbot added a comment.Jan 31 2024, 8:01 AM

Mentioned in SAL (#wikimedia-operations) [2024-01-31T08:01:34Z] <marostegui@cumin1002> dbctl commit (dc=all): 'db2114 (re)pooling @ 50%: After Bookworm upgrade T354506', diff saved to https://phabricator.wikimedia.org/P55928 and previous config saved to /var/cache/conftool/dbconfig/20240131-080132-root.json

Stashbot added a comment.Jan 31 2024, 8:16 AM

Mentioned in SAL (#wikimedia-operations) [2024-01-31T08:16:38Z] <marostegui@cumin1002> dbctl commit (dc=all): 'db2114 (re)pooling @ 75%: After Bookworm upgrade T354506', diff saved to https://phabricator.wikimedia.org/P55930 and previous config saved to /var/cache/conftool/dbconfig/20240131-081637-root.json

Stashbot added a comment.Jan 31 2024, 8:31 AM

Mentioned in SAL (#wikimedia-operations) [2024-01-31T08:31:43Z] <marostegui@cumin1002> dbctl commit (dc=all): 'db2114 (re)pooling @ 100%: After Bookworm upgrade T354506', diff saved to https://phabricator.wikimedia.org/P55932 and previous config saved to /var/cache/conftool/dbconfig/20240131-083142-root.json

Marostegui closed this task as Resolved.Jan 31 2024, 8:47 AM

Done

Marostegui mentioned this in T356960: Upgrade hosts to MariaDB 10.6.Feb 8 2024, 6:25 AM
Marostegui added a parent task: T356960: Upgrade hosts to MariaDB 10.6.
Maintenance_bot moved this task from In progress to Done on the DBA board.Feb 8 2024, 6:29 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits