Page MenuHomePhabricator
Create Task
Maniphest T330920

VRTS Switchover Process
Open, MediumPublic
Actions

Description

What will the VRTS failover process look like?

These are just random bullet points from unrefined ideas that came up in a meeting:

  • Announce Downtime: IRC etc.
  • Swap Configs
  • Database??
  • Update DNS

https://wikitech.wikimedia.org/wiki/VRT_System/Failover

Details

ProjectBranchLines +/-Subject
operations/puppetproduction+1 -1vrts: Fix issue in install script
operations/puppetproduction+7 -2vrts: use variables in rsyncquickdatacopy
operations/puppetproduction+10 -4vrts: post script cleanup & export variables
operations/puppetproduction+46 -14vrts: separate install & ugprade vrts scripts
Customize query in gerrit

Event Timeline

Arnoldokoth created this task.Mar 1 2023, 6:04 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 1 2023, 6:04 PM
Arnoldokoth updated the task description. (Show Details)Mar 1 2023, 6:06 PM
LSobanski added a subscriber: LSobanski.Mar 8 2023, 3:37 PM
Arnoldokoth moved this task from Incoming to Work in Progress on the serviceops-collab board.Mar 13 2023, 3:50 PM
LSobanski triaged this task as Medium priority.Mar 29 2023, 11:40 AM
LSobanski updated the task description. (Show Details)Mar 29 2023, 2:41 PM
ops-monitoring-bot added a comment.May 8 2023, 8:11 PM

Cookbook cookbooks.sre.ganeti.reimage was started by aokoth@cumin1001 for host vrts1001.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.May 8 2023, 8:36 PM

Cookbook cookbooks.sre.ganeti.reimage started by aokoth@cumin1001 for host vrts1001.eqiad.wmnet with OS bullseye completed:

  • vrts1001 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via gnt-instance
    • Host up (Debian installer)
    • Set boot to disk
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/ganeti/reimage/202305082010_aokoth_3063840_vrts1001.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
gerritbot added a comment.Tue, Jun 6, 5:46 PM

Change 927749 had a related patch set uploaded (by AOkoth; author: AOkoth):

[operations/puppet@production] vrts: separate install & ugprade vrts scripts

https://gerrit.wikimedia.org/r/927749

gerritbot added a project: Patch-For-Review.Tue, Jun 6, 5:46 PM
gerritbot added a comment.Tue, Jun 6, 7:37 PM

Change 927749 merged by AOkoth:

[operations/puppet@production] vrts: separate install & ugprade vrts scripts

https://gerrit.wikimedia.org/r/927749

Maintenance_bot removed a project: Patch-For-Review.Tue, Jun 6, 8:10 PM
Arnoldokoth added a project: Znuny.Wed, Jun 7, 2:41 PM
gerritbot added a comment.Wed, Jun 7, 3:26 PM

Change 928084 had a related patch set uploaded (by AOkoth; author: AOkoth):

[operations/puppet@production] vrts: post script cleanup & export variables

https://gerrit.wikimedia.org/r/928084

gerritbot added a project: Patch-For-Review.Wed, Jun 7, 3:26 PM
gerritbot added a comment.Wed, Jun 7, 7:28 PM

Change 928084 merged by AOkoth:

[operations/puppet@production] vrts: post script cleanup & export variables

https://gerrit.wikimedia.org/r/928084

Maintenance_bot removed a project: Patch-For-Review.Wed, Jun 7, 7:30 PM
gerritbot added a comment.Wed, Jun 7, 7:44 PM

Change 928133 had a related patch set uploaded (by AOkoth; author: AOkoth):

[operations/puppet@production] vrts: Fix issue in install script

https://gerrit.wikimedia.org/r/928133

gerritbot added a project: Patch-For-Review.Wed, Jun 7, 7:44 PM

Change 928133 merged by AOkoth:

[operations/puppet@production] vrts: Fix issue in install script

https://gerrit.wikimedia.org/r/928133

Maintenance_bot removed a project: Patch-For-Review.Wed, Jun 7, 8:11 PM
gerritbot added a comment.Wed, Jun 7, 8:13 PM

Change 928136 had a related patch set uploaded (by AOkoth; author: AOkoth):

[operations/puppet@production] vrts: use variables in rsyncquickdatacopy

https://gerrit.wikimedia.org/r/928136

gerritbot added a project: Patch-For-Review.Wed, Jun 7, 8:13 PM
Content licensed under Creative Commons Attribution-ShareAlike 3.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL