Page MenuHomePhabricator
Create Task
Maniphest T301654

Upgrade m5 to Bullseye
Closed, ResolvedPublic
Actions

Description

  • db2135
  • db2078
  • db1132
  • db1117
  • Reimage and move db1107 from m3 to m5 so it can later become m5 master.
  • Promote db1107 to m5 master: T302190
  • Reimage and move db1132 somewhere else - to be moved to s1 for 10.6 testing: T301879#7759181

Details

SubjectRepoBranchLines +/-
db1107: Move from m3 to m5operations/puppetproduction+6 -5
db2134: Disable notificationsoperations/puppetproduction+1 -0
db2134: Disable notificationsoperations/puppetproduction+1 -0
db1107: Disable notificationsoperations/puppetproduction+1 -0
Customize query in gerrit

Related Objects
Search...

Event Timeline

• Marostegui created this task.Feb 14 2022, 7:08 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 14 2022, 7:08 AM
• Marostegui triaged this task as Medium priority.Feb 14 2022, 7:09 AM
• Marostegui moved this task from Triage to Ready on the DBA board.
gerritbot added a comment.Feb 15 2022, 7:08 AM

Change 762683 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] db1107: Disable notifications

https://gerrit.wikimedia.org/r/762683

gerritbot added a project: Patch-For-Review.Feb 15 2022, 7:08 AM

Change 762683 merged by Marostegui:

[operations/puppet@production] db1107: Disable notifications

https://gerrit.wikimedia.org/r/762683

Maintenance_bot removed a project: Patch-For-Review.Feb 15 2022, 7:10 AM
• Marostegui mentioned this in T301219: Switchover m3 master (db1107 -> db1183).Feb 15 2022, 8:03 AM
• Marostegui mentioned this in T300835: Upgrade m3 to Bullseye.
• Marostegui moved this task from Ready to In progress on the DBA board.Feb 15 2022, 8:06 AM
gerritbot added a comment.Feb 15 2022, 8:09 AM

Change 762739 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] db2134: Disable notifications

https://gerrit.wikimedia.org/r/762739

gerritbot added a project: Patch-For-Review.Feb 15 2022, 8:09 AM

Change 762739 merged by Marostegui:

[operations/puppet@production] db2134: Disable notifications

https://gerrit.wikimedia.org/r/762739

gerritbot added a comment.Feb 15 2022, 8:12 AM

Change 762741 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] db2134: Disable notifications

https://gerrit.wikimedia.org/r/762741

gerritbot added a comment.Feb 15 2022, 8:14 AM

Change 762741 merged by Marostegui:

[operations/puppet@production] db2134: Disable notifications

https://gerrit.wikimedia.org/r/762741

ops-monitoring-bot added a comment.Feb 15 2022, 8:15 AM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1001 for host db2135.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Feb 15 2022, 8:44 AM

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1001 for host db2135.codfw.wmnet with OS bullseye completed:

  • db2135 (PASS)
    • Downtimed on Icinga
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202202150814_marostegui_8983_db2135.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
• Marostegui updated the task description. (Show Details)Feb 15 2022, 8:46 AM
• Marostegui added a comment.Feb 15 2022, 9:01 AM

db1107 is no longer m3 master - let's wait 24 after the switch to move it to m5

Maintenance_bot removed a project: Patch-For-Review.Feb 15 2022, 9:10 AM
gerritbot added a comment.Feb 21 2022, 6:18 AM

Change 764111 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] db1107: Move from m3 to m5

https://gerrit.wikimedia.org/r/764111

gerritbot added a project: Patch-For-Review.Feb 21 2022, 6:18 AM
ops-monitoring-bot added a comment.Feb 21 2022, 6:20 AM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1001 for host db1107.eqiad.wmnet with OS bullseye

gerritbot added a comment.Feb 21 2022, 6:20 AM

Change 764111 merged by Marostegui:

[operations/puppet@production] db1107: Move from m3 to m5

https://gerrit.wikimedia.org/r/764111

• Marostegui mentioned this in P21094 Masterwork From Distant Lands.Feb 21 2022, 6:42 AM
ops-monitoring-bot added a comment.Feb 21 2022, 6:46 AM

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1001 for host db1107.eqiad.wmnet with OS bullseye completed:

  • db1107 (WARN)
    • Downtimed on Icinga
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202202210620_marostegui_2167_db1107.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
• Marostegui updated the task description. (Show Details)Feb 21 2022, 6:51 AM
• Marostegui updated the task description. (Show Details)Mar 9 2022, 8:28 AM
• Marostegui closed this task as Resolved.Mar 9 2022, 2:10 PM
• Marostegui closed subtask T302190: Switchover m5 master (db1132 -> db1107) as Resolved.
• Marostegui updated the task description. (Show Details)

This was all done - tomorrow I will reimage db1132 to Bullseye and move it to s1 for 10.6 testing: T301879#7759181
Closing this task

Maintenance_bot moved this task from In progress to Done on the DBA board.Mar 9 2022, 2:15 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL