Page MenuHomePhabricator
Create Task
Maniphest T358080

Upgrade es2 to MariaDB 10.6
Closed, ResolvedPublic
Actions

Description

Also reimage to Bookworm

  • es2033
  • es2026 (this won't boot via PXE T357951)
  • es2031
  • es1033
  • es1026
  • es1030

Details

Related Changes in Gerrit:
SubjectAuthorRepoBranchLines +/-
wmnet: Promote es2026 to es2 masterMarosteguioperations/dnsmaster+1 -1
es1033: Disable notificationsMarosteguioperations/puppetproduction+1 -0
es2031: Disable notificationsMarosteguioperations/puppetproduction+1 -0
es1030: Disable notificationsMarosteguioperations/puppetproduction+1 -0
es1026: Disable notificationsMarosteguioperations/puppetproduction+1 -0
Customize query in gerrit

Related Objects
Search...

Event Timeline

Marostegui created this task.Feb 21 2024, 5:36 AM
Restricted Application added a subscriber: • Aklapper. · View Herald TranscriptFeb 21 2024, 5:36 AM
Marostegui triaged this task as Medium priority.Feb 21 2024, 5:36 AM
Marostegui moved this task from Triage to In progress on the DBA board.
Marostegui mentioned this in T356960: Upgrade hosts to MariaDB 10.6.
gerritbot added a comment.Feb 21 2024, 5:37 AM

Change 1005221 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] es1026: Disable notifications

https://gerrit.wikimedia.org/r/1005221

gerritbot added a project: Patch-For-Review.Feb 21 2024, 5:37 AM
Stashbot added a comment.Feb 21 2024, 5:38 AM

Mentioned in SAL (#wikimedia-operations) [2024-02-21T05:38:22Z] <marostegui@cumin1002> dbctl commit (dc=all): 'Depool es1026 T358080', diff saved to https://phabricator.wikimedia.org/P57434 and previous config saved to /var/cache/conftool/dbconfig/20240221-053822-root.json

gerritbot added a comment.Feb 21 2024, 5:39 AM

Change 1005221 merged by Marostegui:

[operations/puppet@production] es1026: Disable notifications

https://gerrit.wikimedia.org/r/1005221

ops-monitoring-bot added a comment.Feb 21 2024, 5:39 AM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1002 for host es1026.eqiad.wmnet with OS bookworm

Marostegui updated the task description. (Show Details)Feb 21 2024, 6:09 AM
ops-monitoring-bot added a comment.Feb 21 2024, 6:11 AM

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1002 for host es1026.eqiad.wmnet with OS bookworm completed:

  • es1026 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202402210555_marostegui_1202635_es1026.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
Maintenance_bot removed a project: Patch-For-Review.Feb 21 2024, 6:30 AM
gerritbot added a comment.Feb 21 2024, 9:09 AM

Change 1005439 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] es1030: Disable notifications

https://gerrit.wikimedia.org/r/1005439

gerritbot added a project: Patch-For-Review.Feb 21 2024, 9:09 AM
Stashbot added a comment.Feb 21 2024, 9:09 AM

Mentioned in SAL (#wikimedia-operations) [2024-02-21T09:09:58Z] <marostegui@cumin1002> dbctl commit (dc=all): 'Depool es1030 T358080', diff saved to https://phabricator.wikimedia.org/P57474 and previous config saved to /var/cache/conftool/dbconfig/20240221-090957-root.json

ops-monitoring-bot added a comment.Feb 21 2024, 9:10 AM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1002 for host es1030.eqiad.wmnet with OS bookworm

gerritbot added a comment.Feb 21 2024, 9:11 AM

Change 1005439 merged by Marostegui:

[operations/puppet@production] es1030: Disable notifications

https://gerrit.wikimedia.org/r/1005439

Maintenance_bot removed a project: Patch-For-Review.Feb 21 2024, 9:30 AM
Marostegui updated the task description. (Show Details)Feb 21 2024, 9:39 AM
ops-monitoring-bot added a comment.Feb 21 2024, 9:42 AM

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1002 for host es1030.eqiad.wmnet with OS bookworm completed:

  • es1030 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202402210926_marostegui_1233612_es1030.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
Stashbot added a comment.Feb 21 2024, 9:52 AM

Mentioned in SAL (#wikimedia-operations) [2024-02-21T09:52:06Z] <marostegui@cumin1002> dbctl commit (dc=all): 'Depool es2031 T358080', diff saved to https://phabricator.wikimedia.org/P57487 and previous config saved to /var/cache/conftool/dbconfig/20240221-095205-root.json

gerritbot added a comment.Feb 21 2024, 9:53 AM

Change 1005448 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] es2031: Disable notifications

https://gerrit.wikimedia.org/r/1005448

gerritbot added a project: Patch-For-Review.Feb 21 2024, 9:53 AM
ops-monitoring-bot added a comment.Feb 21 2024, 9:53 AM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1002 for host es2031.codfw.wmnet with OS bookworm

gerritbot added a comment.Feb 21 2024, 9:54 AM

Change 1005448 merged by Marostegui:

[operations/puppet@production] es2031: Disable notifications

https://gerrit.wikimedia.org/r/1005448

Maintenance_bot removed a project: Patch-For-Review.Feb 21 2024, 10:30 AM
ops-monitoring-bot added a comment.Feb 21 2024, 10:35 AM

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1002 for host es2031.codfw.wmnet with OS bookworm completed:

  • es2031 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202402211014_marostegui_1240054_es2031.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
Marostegui updated the task description. (Show Details)Feb 21 2024, 10:45 AM
Stashbot added a comment.Feb 21 2024, 12:02 PM

Mentioned in SAL (#wikimedia-operations) [2024-02-21T12:02:03Z] <marostegui@cumin1002> dbctl commit (dc=all): 'Depool es2026 T358080', diff saved to https://phabricator.wikimedia.org/P57524 and previous config saved to /var/cache/conftool/dbconfig/20240221-120202-root.json

ops-monitoring-bot added a comment.Feb 21 2024, 12:03 PM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1002 for host es2026.codfw.wmnet with OS bookworm

ops-monitoring-bot added a comment.Feb 21 2024, 12:15 PM

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1002 for host es2026.codfw.wmnet with OS bookworm executed with errors:

  • es2026 (FAIL)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details,You can also try typing "install-console" es2026.codfw.wmnet to get a root shellbut depending on the failure this may not work.
ops-monitoring-bot added a comment.Feb 21 2024, 12:15 PM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1002 for host es2026.codfw.wmnet with OS bookworm

Marostegui updated the task description. (Show Details)Feb 21 2024, 12:20 PM
Marostegui added a subtask: T357951: db2137 and es2026 don't get an IP via PXE boot.
ops-monitoring-bot added a comment.Feb 21 2024, 12:21 PM

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1002 for host es2026.codfw.wmnet with OS bookworm executed with errors:

  • es2026 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details,You can also try typing "install-console" es2026.codfw.wmnet to get a root shellbut depending on the failure this may not work.
Marostegui updated the task description. (Show Details)Feb 21 2024, 3:12 PM
Marostegui closed subtask T357951: db2137 and es2026 don't get an IP via PXE boot as Resolved.Feb 21 2024, 4:10 PM
Stashbot added a comment.Feb 22 2024, 6:42 AM

Mentioned in SAL (#wikimedia-operations) [2024-02-22T06:42:06Z] <marostegui@cumin1002> dbctl commit (dc=all): 'Promote es1030 as es2 master T358080', diff saved to https://phabricator.wikimedia.org/P57651 and previous config saved to /var/cache/conftool/dbconfig/20240222-064205-marostegui.json

gerritbot added a comment.Feb 22 2024, 6:42 AM

Change 1005649 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] es1033: Disable notifications

https://gerrit.wikimedia.org/r/1005649

gerritbot added a project: Patch-For-Review.Feb 22 2024, 6:42 AM
Stashbot added a comment.Feb 22 2024, 6:42 AM

Mentioned in SAL (#wikimedia-operations) [2024-02-22T06:42:53Z] <marostegui@cumin1002> dbctl commit (dc=all): 'Depool es1033 T358080', diff saved to https://phabricator.wikimedia.org/P57652 and previous config saved to /var/cache/conftool/dbconfig/20240222-064253-root.json

ops-monitoring-bot added a comment.Feb 22 2024, 6:44 AM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1002 for host es1033.eqiad.wmnet with OS bookworm

gerritbot added a comment.Feb 22 2024, 6:44 AM

Change 1005649 merged by Marostegui:

[operations/puppet@production] es1033: Disable notifications

https://gerrit.wikimedia.org/r/1005649

ops-monitoring-bot added a comment.Feb 22 2024, 7:19 AM

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1002 for host es1033.eqiad.wmnet with OS bookworm completed:

  • es1033 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Unable to downtime the new host on Icinga/Alertmanager, the sre.hosts.downtime cookbook returned 99
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202402220658_marostegui_1410956_es1033.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
Marostegui updated the task description. (Show Details)Feb 22 2024, 7:27 AM
Stashbot added a comment.Feb 22 2024, 7:30 AM

Mentioned in SAL (#wikimedia-operations) [2024-02-22T07:30:17Z] <marostegui@cumin1002> dbctl commit (dc=all): 'Promote es2026 as es2 codfw master T358080', diff saved to https://phabricator.wikimedia.org/P57657 and previous config saved to /var/cache/conftool/dbconfig/20240222-073017-marostegui.json

Maintenance_bot removed a project: Patch-For-Review.Feb 22 2024, 7:30 AM
gerritbot added a comment.Feb 22 2024, 7:35 AM

Change 1005653 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/dns@master] wmnet: Promote es2026 to es2 master

https://gerrit.wikimedia.org/r/1005653

gerritbot added a project: Patch-For-Review.Feb 22 2024, 7:35 AM
gerritbot added a comment.Feb 22 2024, 7:39 AM

Change 1005653 merged by Marostegui:

[operations/dns@master] wmnet: Promote es2026 to es2 master

https://gerrit.wikimedia.org/r/1005653

Stashbot added a comment.Feb 22 2024, 7:40 AM

Mentioned in SAL (#wikimedia-operations) [2024-02-22T07:40:42Z] <marostegui@cumin1002> dbctl commit (dc=all): 'Depool es2033 T358080', diff saved to https://phabricator.wikimedia.org/P57659 and previous config saved to /var/cache/conftool/dbconfig/20240222-074042-root.json

ops-monitoring-bot added a comment.Feb 22 2024, 7:54 AM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1002 for host es2033.codfw.wmnet with OS bookworm

Marostegui updated the task description. (Show Details)Feb 22 2024, 8:27 AM
Maintenance_bot removed a project: Patch-For-Review.Feb 22 2024, 8:30 AM
ops-monitoring-bot added a comment.Feb 22 2024, 8:30 AM

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1002 for host es2033.codfw.wmnet with OS bookworm completed:

  • es2033 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202402220814_marostegui_1420006_es2033.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
Marostegui closed this task as Resolved.Feb 22 2024, 10:17 AM
Maintenance_bot moved this task from In progress to Done on the DBA board.Feb 22 2024, 10:29 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits