Page MenuHomePhabricator
Create Task
Maniphest T299344

Upgrade m1 to Bullseye
Closed, ResolvedPublic
Actions

Description

Let's upgrade m1 to Bullseye.
This will have our first master (apart from parsercache) running Bullseye.

Hosts:

  • db1159 (master)
  • db1117
  • db2132 (codfw master)
  • db2078

Let's use db1128 (once the s1 tests are done) as a master to failover to.

  • Move db1128 to m1 as a future master
  • Switchover m1 master, from db1159 to db1128 T299624

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
install_server: Do not format db1128operations/puppetproduction+1 -1
mariadb: Move db1128 to m1operations/puppetproduction+9 -9
instances.yaml: Remove db1128 from dbctloperations/puppetproduction+0 -1
db1117: Disable notificationsoperations/puppetproduction+1 -0
db2132: Disable notificationsoperations/puppetproduction+1 -0
Customize query in gerrit

Related Objects
Search...

Event Timeline

Marostegui created this task.Jan 17 2022, 1:01 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 17 2022, 1:01 PM
Marostegui triaged this task as Medium priority.Jan 17 2022, 1:01 PM
Marostegui moved this task from Triage to Ready on the DBA board.
Marostegui added a subscriber: • Kormat.Jan 17 2022, 1:23 PM

@Kormat heads up, I am planning to switch this master probably next week if all goes fine. In case you want to work on db-switchover to test that it doesn't upgrade tendril anymore (which is something we need to do before we can stop mysql on tendril's host: T297605)

Marostegui updated the task description. (Show Details)Jan 17 2022, 1:24 PM
gerritbot added a comment.Jan 17 2022, 2:14 PM

Change 754511 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] db2132: Disable notifications

https://gerrit.wikimedia.org/r/754511

gerritbot added a project: Patch-For-Review.Jan 17 2022, 2:14 PM
Stashbot added a comment.Jan 17 2022, 2:15 PM

Mentioned in SAL (#wikimedia-operations) [2022-01-17T14:15:37Z] <marostegui> Reimage db2132 to Bullseye T299344

gerritbot added a comment.Jan 17 2022, 2:15 PM

Change 754511 merged by Marostegui:

[operations/puppet@production] db2132: Disable notifications

https://gerrit.wikimedia.org/r/754511

ops-monitoring-bot added a comment.Jan 17 2022, 2:16 PM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1001 for host db2132.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Jan 17 2022, 2:50 PM

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1001 for host db2132.codfw.wmnet with OS bullseye completed:

  • db2132 (WARN)
    • Downtimed on Icinga
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202201171416_marostegui_2982_db2132.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
Marostegui updated the task description. (Show Details)Jan 17 2022, 2:51 PM

I will use db1128 as a floating host. It needs to be removed from s1 once the successful live MW tests are done, which I expect to be during this week.

Maintenance_bot removed a project: Patch-For-Review.Jan 17 2022, 3:10 PM
Marostegui moved this task from Ready to In progress on the DBA board.Jan 18 2022, 10:29 AM
gerritbot added a comment.Jan 18 2022, 10:30 AM

Change 754878 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] db1117: Disable notifications

https://gerrit.wikimedia.org/r/754878

gerritbot added a project: Patch-For-Review.Jan 18 2022, 10:30 AM

Change 754878 merged by Marostegui:

[operations/puppet@production] db1117: Disable notifications

https://gerrit.wikimedia.org/r/754878

ops-monitoring-bot added a comment.Jan 18 2022, 10:31 AM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1001 for host db1117.eqiad.wmnet with OS bullseye

Marostegui updated the task description. (Show Details)Jan 18 2022, 10:51 AM
ops-monitoring-bot added a comment.Jan 18 2022, 10:56 AM

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1001 for host db1117.eqiad.wmnet with OS bullseye completed:

  • db1117 (WARN)
    • Downtimed on Icinga
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202201181031_marostegui_3001_db1117.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
Marostegui updated the task description. (Show Details)Jan 18 2022, 10:57 AM
Maintenance_bot removed a project: Patch-For-Review.Jan 18 2022, 11:10 AM
gerritbot added a comment.Jan 19 2022, 12:55 PM

Change 755353 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] instances.yaml: Remove db1128 from dbctl

https://gerrit.wikimedia.org/r/755353

gerritbot added a project: Patch-For-Review.Jan 19 2022, 12:55 PM

Change 755353 merged by Marostegui:

[operations/puppet@production] instances.yaml: Remove db1128 from dbctl

https://gerrit.wikimedia.org/r/755353

Stashbot added a comment.Jan 19 2022, 12:56 PM

Mentioned in SAL (#wikimedia-operations) [2022-01-19T12:56:59Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Remove db1128 from dbctl T299344', diff saved to https://phabricator.wikimedia.org/P18858 and previous config saved to /var/cache/conftool/dbconfig/20220119-125658-marostegui.json

Marostegui added a comment.Jan 19 2022, 12:57 PM

db1128 removed from dbctl (it was serving in s1) and ready to be moved to m1.

Maintenance_bot removed a project: Patch-For-Review.Jan 19 2022, 1:10 PM
gerritbot added a comment.Jan 20 2022, 7:22 AM

Change 755526 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] mariadb: Move db1128 to m1

https://gerrit.wikimedia.org/r/755526

gerritbot added a project: Patch-For-Review.Jan 20 2022, 7:22 AM

Change 755526 merged by Marostegui:

[operations/puppet@production] mariadb: Move db1128 to m1

https://gerrit.wikimedia.org/r/755526

ops-monitoring-bot added a comment.Jan 20 2022, 7:32 AM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1001 for host db1128.eqiad.wmnet with OS bullseye

Marostegui changed the status of subtask T299624: Switchover m1 master (db1159 -> db1128) from Open to Stalled.Jan 20 2022, 7:56 AM
Stashbot added a comment.Jan 20 2022, 7:57 AM

Mentioned in SAL (#wikimedia-operations) [2022-01-20T07:57:14Z] <marostegui> Stop mysql on db1117 to clone db1128 T299344

ops-monitoring-bot added a comment.Jan 20 2022, 7:59 AM

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1001 for host db1128.eqiad.wmnet with OS bullseye completed:

  • db1128 (WARN)
    • Downtimed on Icinga
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202201200732_marostegui_10495_db1128.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
gerritbot added a comment.Jan 20 2022, 8:02 AM

Change 755530 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] install_server: Do not format db1128

https://gerrit.wikimedia.org/r/755530

gerritbot added a comment.Jan 20 2022, 8:03 AM

Change 755530 merged by Marostegui:

[operations/puppet@production] install_server: Do not format db1128

https://gerrit.wikimedia.org/r/755530

Maintenance_bot removed a project: Patch-For-Review.Jan 20 2022, 8:10 AM
Marostegui added a comment.Jan 20 2022, 9:38 AM

db1128 is now replicating in m1 with Bullseye.

Marostegui updated the task description. (Show Details)Jan 20 2022, 9:38 AM
Marostegui changed the status of subtask T299624: Switchover m1 master (db1159 -> db1128) from Stalled to Open.Jan 21 2022, 11:28 AM
Marostegui updated the task description. (Show Details)Jan 21 2022, 11:54 AM
Marostegui closed subtask T299624: Switchover m1 master (db1159 -> db1128) as Resolved.Jan 27 2022, 10:31 AM
Marostegui updated the task description. (Show Details)
Marostegui closed this task as Resolved.Jan 27 2022, 10:33 AM

All done. db1159 will be moved to m2: T300243

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits