⚓ T361543 Upgrade s2 to MariaDB 10.6

Subject	Repo	Branch	Lines +/-
db1162: Disable notifications	operations/puppet	production	+2 -1
db1156: Migrate to MariaDB 10.6	operations/puppet	production	+1 -0
db1222: Upgrade to Bookworm and MariaDB 10.6	operations/puppet	production	+1 -0
db1229: Disable notifications	operations/puppet	production	+1 -0

Status	Subtype	Assigned	Task
Open		None	T356960 Upgrade hosts to MariaDB 10.6
Resolved		Marostegui	T361543 Upgrade s2 to MariaDB 10.6
Resolved	Request	Jhancock.wm	T361779 decommission db2104.codfw.wmnet
Duplicate		None	T361780 Switchover s2 master (db2107 -> db2204)
Resolved		Marostegui	T362036 Switchover s2 master (db1162 -> db1222)

Marostegui created this task.Tue, Apr 2, 5:42 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptTue, Apr 2, 5:42 AM

Marostegui triaged this task as Medium priority.Tue, Apr 2, 5:43 AM

Marostegui moved this task from Triage to In progress on the DBA board.

Marostegui mentioned this in T356960: Upgrade hosts to MariaDB 10.6.

Mentioned in SAL (#wikimedia-operations) [2024-04-02T05:44:09Z] <marostegui@cumin1002> dbctl commit (dc=all): 'Depool db1229 T361543', diff saved to https://phabricator.wikimedia.org/P59116 and previous config saved to /var/cache/conftool/dbconfig/20240402-054408-root.json

Change #1016080 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] db1229: Disable notifications

https://gerrit.wikimedia.org/r/1016080

gerritbot added a project: Patch-For-Review.Tue, Apr 2, 5:45 AM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1002 for host db1229.eqiad.wmnet with OS bookworm

Change #1016080 merged by Marostegui:

[operations/puppet@production] db1229: Disable notifications

https://gerrit.wikimedia.org/r/1016080

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1002 for host db1229.eqiad.wmnet with OS bookworm completed:

db1229 (WARN)
- Downtimed on Icinga/Alertmanager
- Disabled Puppet
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Add puppet_version metadata to Debian installer
- Checked BIOS boot parameters are back to normal
- Host up (new fresh bookworm OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- Removed previous downtime on Alertmanager (old OS)
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202404020603_marostegui_198803_db1229.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is not optimal, downtime not removed
- Updated Netbox data from PuppetDB

Marostegui updated the task description. (Show Details)Tue, Apr 2, 6:24 AM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1002 for host db1197.eqiad.wmnet with OS bookworm

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1002 for host db1197.eqiad.wmnet with OS bookworm completed:

db1197 (WARN)
- Downtimed on Icinga/Alertmanager
- Disabled Puppet
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Add puppet_version metadata to Debian installer
- Checked BIOS boot parameters are back to normal
- Host up (new fresh bookworm OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- Removed previous downtime on Alertmanager (old OS)
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202404020832_marostegui_221043_db1197.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is not optimal, downtime not removed
- Updated Netbox data from PuppetDB
- Cleared switch DHCP cache and MAC table for the host IP and MAC (EVPN Switch)

Marostegui updated the task description. (Show Details)Tue, Apr 2, 8:58 AM

Mentioned in SAL (#wikimedia-operations) [2024-04-02T12:04:57Z] <marostegui@cumin1002> dbctl commit (dc=all): 'Depool db1188 T361543', diff saved to https://phabricator.wikimedia.org/P59158 and previous config saved to /var/cache/conftool/dbconfig/20240402-120455-root.json

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1002 for host db1188.eqiad.wmnet with OS bookworm

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1002 for host db1188.eqiad.wmnet with OS bookworm completed:

db1188 (WARN)
- Downtimed on Icinga/Alertmanager
- Disabled Puppet
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Add puppet_version metadata to Debian installer
- Checked BIOS boot parameters are back to normal
- Host up (new fresh bookworm OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- Removed previous downtime on Alertmanager (old OS)
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202404021222_marostegui_258065_db1188.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is not optimal, downtime not removed
- Updated Netbox data from PuppetDB

Marostegui updated the task description. (Show Details)Tue, Apr 2, 12:45 PM

Change #1016489 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] db1222: Upgrade to Bookworm and MariaDB 10.6

https://gerrit.wikimedia.org/r/1016489

Mentioned in SAL (#wikimedia-operations) [2024-04-03T05:11:50Z] <marostegui@cumin1002> dbctl commit (dc=all): 'Depool db1222 T361543', diff saved to https://phabricator.wikimedia.org/P59239 and previous config saved to /var/cache/conftool/dbconfig/20240403-051149-root.json

Change #1016489 merged by Marostegui:

[operations/puppet@production] db1222: Upgrade to Bookworm and MariaDB 10.6

https://gerrit.wikimedia.org/r/1016489

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1002 for host db1222.eqiad.wmnet with OS bookworm

Marostegui updated the task description. (Show Details)Wed, Apr 3, 5:41 AM

Mentioned in SAL (#wikimedia-operations) [2024-04-03T05:43:11Z] <marostegui@cumin1002> dbctl commit (dc=all): 'Depool db2148 T361543', diff saved to https://phabricator.wikimedia.org/P59240 and previous config saved to /var/cache/conftool/dbconfig/20240403-054310-root.json

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1002 for host db2148.codfw.wmnet with OS bookworm

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1002 for host db1222.eqiad.wmnet with OS bookworm completed:

db1222 (WARN)
- Downtimed on Icinga/Alertmanager
- Disabled Puppet
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Add puppet_version metadata to Debian installer
- Checked BIOS boot parameters are back to normal
- Host up (new fresh bookworm OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- Removed previous downtime on Alertmanager (old OS)
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202404030528_marostegui_400270_db1222.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is not optimal, downtime not removed
- Updated Netbox data from PuppetDB

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1002 for host db2148.codfw.wmnet with OS bookworm completed:

db2148 (WARN)
- Downtimed on Icinga/Alertmanager
- Disabled Puppet
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Add puppet_version metadata to Debian installer
- Checked BIOS boot parameters are back to normal
- Host up (new fresh bookworm OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- Removed previous downtime on Alertmanager (old OS)
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202404030604_marostegui_405823_db2148.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is not optimal, downtime not removed
- Updated Netbox data from PuppetDB
- Cleared switch DHCP cache and MAC table for the host IP and MAC (EVPN Switch)

Marostegui updated the task description. (Show Details)Wed, Apr 3, 6:40 AM

Mentioned in SAL (#wikimedia-operations) [2024-04-03T07:09:47Z] <marostegui@cumin1002> dbctl commit (dc=all): 'Depool db2125 T361543', diff saved to https://phabricator.wikimedia.org/P59256 and previous config saved to /var/cache/conftool/dbconfig/20240403-070946-root.json

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1002 for host db2125.codfw.wmnet with OS bookworm

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1002 for host db2125.codfw.wmnet with OS bookworm completed:

db2125 (WARN)
- Downtimed on Icinga/Alertmanager
- Disabled Puppet
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Add puppet_version metadata to Debian installer
- Checked BIOS boot parameters are back to normal
- Host up (new fresh bookworm OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- Removed previous downtime on Alertmanager (old OS)
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202404030735_marostegui_418774_db2125.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is not optimal, downtime not removed
- Updated Netbox data from PuppetDB

Marostegui updated the task description. (Show Details)Wed, Apr 3, 8:01 AM

Mentioned in SAL (#wikimedia-operations) [2024-04-04T05:17:59Z] <marostegui@cumin1002> dbctl commit (dc=all): 'Depool db2126 T361543', diff saved to https://phabricator.wikimedia.org/P59407 and previous config saved to /var/cache/conftool/dbconfig/20240404-051758-root.json

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1002 for host db2126.codfw.wmnet with OS bookworm

Marostegui updated the task description. (Show Details)Thu, Apr 4, 5:59 AM

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1002 for host db2126.codfw.wmnet with OS bookworm completed:

db2126 (WARN)
- Downtimed on Icinga/Alertmanager
- Disabled Puppet
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Add puppet_version metadata to Debian installer
- Checked BIOS boot parameters are back to normal
- Host up (new fresh bookworm OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- Removed previous downtime on Alertmanager (old OS)
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202404040539_marostegui_597454_db2126.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is not optimal, downtime not removed
- Updated Netbox data from PuppetDB

Marostegui updated the task description. (Show Details)Thu, Apr 4, 7:27 AM

Marostegui updated the task description. (Show Details)Thu, Apr 4, 7:35 AM

Marostegui added a subtask: T361779: decommission db2104.codfw.wmnet.

Marostegui added a subtask: T361780: Switchover s2 master (db2107 -> db2204).

Marostegui updated the task description. (Show Details)Thu, Apr 4, 9:36 AM

Jhancock.wm closed subtask T361779: decommission db2104.codfw.wmnet as Resolved.Thu, Apr 4, 4:27 PM

Change #1017535 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] db1156: Migrate to MariaDB 10.6

https://gerrit.wikimedia.org/r/1017535

Change #1017535 merged by Marostegui:

[operations/puppet@production] db1156: Migrate to MariaDB 10.6

https://gerrit.wikimedia.org/r/1017535

Marostegui updated the task description. (Show Details)Mon, Apr 8, 6:10 AM

Marostegui added a subtask: T362036: Switchover s2 master (db1162 -> db1222).

Marostegui closed subtask T362036: Switchover s2 master (db1162 -> db1222) as Resolved.Tue, Apr 9, 5:32 AM

Change #1017965 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] db1162: Disable notifications

https://gerrit.wikimedia.org/r/1017965

Change #1017965 merged by Marostegui:

[operations/puppet@production] db1162: Disable notifications

https://gerrit.wikimedia.org/r/1017965

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1002 for host db1162.eqiad.wmnet with OS bookworm

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1002 for host db1162.eqiad.wmnet with OS bookworm completed:

db1162 (WARN)
- Downtimed on Icinga/Alertmanager
- Disabled Puppet
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Add puppet_version metadata to Debian installer
- Checked BIOS boot parameters are back to normal
- Host up (new fresh bookworm OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- Removed previous downtime on Alertmanager (old OS)
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202404090548_marostegui_1502589_db1162.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is not optimal, downtime not removed
- Updated Netbox data from PuppetDB

All core hosts done.

Maintenance_bot moved this task from In progress to Done on the DBA board.Tue, Apr 9, 7:29 AM

Upgrade s2 to MariaDB 10.6
Closed, ResolvedPublic
Actions

Description

Details

Related Objects
Search...

Event Timeline

Upgrade s2 to MariaDB 10.6Closed, ResolvedPublicActions

Description

Details

Related ObjectsSearch...

Event Timeline

Upgrade s2 to MariaDB 10.6
Closed, ResolvedPublic
Actions

Related Objects
Search...