Page MenuHomePhabricator
Create Task
Maniphest T300243

Upgrade m2 to Bullseye
Closed, ResolvedPublic
Actions

Description

Let's upgrade m2 to Bullseye.

  • db2133
  • db2078
  • db1183
  • db1117
  • Reimage and move db1159 from m1 to m2 so it can later become m2 master.
  • Promote db1159 to m2 master: T300329

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
mariadb: Move db1183 from m2 to m3operations/puppetproduction+7 -6
db1183: Disable notificationsoperations/puppetproduction+1 -0
db2133: Disable notificationsoperations/puppetproduction+1 -0
db1159: Move it to m2operations/puppetproduction+6 -6
Customize query in gerrit

Related Objects
Search...

Event Timeline

Marostegui created this task.Jan 27 2022, 10:31 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 27 2022, 10:31 AM
Marostegui triaged this task as Medium priority.Jan 27 2022, 10:31 AM
Marostegui moved this task from Triage to In progress on the DBA board.
Marostegui mentioned this in T299624: Switchover m1 master (db1159 -> db1128).
Marostegui updated the task description. (Show Details)Jan 27 2022, 10:33 AM
Marostegui mentioned this in T299344: Upgrade m1 to Bullseye.
Marostegui updated the task description. (Show Details)
gerritbot added a comment.Jan 27 2022, 10:36 AM

Change 757626 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] db1159: Move it to m2

https://gerrit.wikimedia.org/r/757626

gerritbot added a project: Patch-For-Review.Jan 27 2022, 10:36 AM
Marostegui updated the task description. (Show Details)Jan 27 2022, 10:37 AM
ops-monitoring-bot added a comment.Jan 27 2022, 10:38 AM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1001 for host db1159.eqiad.wmnet with OS bullseye

gerritbot added a comment.Jan 27 2022, 10:39 AM

Change 757626 merged by Marostegui:

[operations/puppet@production] db1159: Move it to m2

https://gerrit.wikimedia.org/r/757626

Marostegui updated the task description. (Show Details)Jan 27 2022, 10:43 AM
ops-monitoring-bot added a comment.Jan 27 2022, 11:09 AM

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1001 for host db1159.eqiad.wmnet with OS bullseye completed:

  • db1159 (WARN)
    • Downtimed on Icinga
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202201271038_marostegui_11095_db1159.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
Maintenance_bot removed a project: Patch-For-Review.Jan 27 2022, 11:10 AM
Marostegui added a comment.Jan 27 2022, 1:52 PM

db1159 is now replicating on m2.

Marostegui updated the task description. (Show Details)Jan 27 2022, 1:53 PM
gerritbot added a comment.Jan 28 2022, 6:08 AM

Change 757797 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] db2133: Disable notifications

https://gerrit.wikimedia.org/r/757797

gerritbot added a project: Patch-For-Review.Jan 28 2022, 6:08 AM

Change 757797 merged by Marostegui:

[operations/puppet@production] db2133: Disable notifications

https://gerrit.wikimedia.org/r/757797

Maintenance_bot removed a project: Patch-For-Review.Jan 28 2022, 6:10 AM
ops-monitoring-bot added a comment.Jan 28 2022, 6:11 AM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1001 for host db2133.codfw.wmnet with OS bullseye

Marostegui updated the task description. (Show Details)Jan 28 2022, 6:41 AM
ops-monitoring-bot added a comment.Jan 28 2022, 6:42 AM

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1001 for host db2133.codfw.wmnet with OS bullseye completed:

  • db2133 (PASS)
    • Downtimed on Icinga
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202201280611_marostegui_3180_db2133.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
Marostegui updated the task description. (Show Details)Jan 28 2022, 6:54 AM
Marostegui added a comment.Jan 28 2022, 10:25 AM

Upgraded db1159 in zarcillo

Marostegui closed subtask T300329: Switchover m2 master (db1183 -> db1159) as Resolved.Feb 3 2022, 9:17 AM
Marostegui updated the task description. (Show Details)Feb 3 2022, 9:24 AM
gerritbot added a comment.Feb 3 2022, 9:25 AM

Change 759449 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] db1183: Disable notifications

https://gerrit.wikimedia.org/r/759449

gerritbot added a project: Patch-For-Review.Feb 3 2022, 9:25 AM

Change 759449 merged by Marostegui:

[operations/puppet@production] db1183: Disable notifications

https://gerrit.wikimedia.org/r/759449

gerritbot added a comment.Feb 3 2022, 9:29 AM

Change 759450 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] mariadb: Move db1183 from m2 to m3

https://gerrit.wikimedia.org/r/759450

gerritbot added a comment.Feb 3 2022, 9:30 AM

Change 759450 merged by Marostegui:

[operations/puppet@production] mariadb: Move db1183 from m2 to m3

https://gerrit.wikimedia.org/r/759450

Marostegui closed this task as Resolved.Feb 3 2022, 9:31 AM

db1183 is being reimaged and moved to m3.

Maintenance_bot removed a project: Patch-For-Review.Feb 3 2022, 10:10 AM
Marostegui updated the task description. (Show Details)Feb 3 2022, 11:11 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits