Page MenuHomePhabricator
Create Task
Maniphest T253124

Upgrade cloudnet2003 and cloudnet2004 to Debian Buster
Closed, ResolvedPublic
Actions

Description

I just did these upgrades in codfw1dev and things went very smoothly. Steps:

  • Merge an install_server patch moving these hosts to Buster
  • Determine which host is active and which is standby:
root@cloudcontrol1003:~# neutron l3-agent-list-hosting-router d93771ba-2711-4f88-804a-8df6fd03978a
  • Re-image standby host, confirm puppet stability, reboot
  • Confirm (with above command) that the standby node is up and running
  • Fail-over by stopping the l3-agent on the active host
service neutron-l3-agent stop
  • Confirm networking still working, confirm that active/standby hosts have traded places
  • Re-image remaining host

Details

SubjectRepoBranchLines +/-
cloudnet1003 and cloudnet1004: next install with Busteroperations/puppetproduction+0 -2
Customize query in gerrit

Related Objects
Search...

Event Timeline

Andrew created this task.May 19 2020, 2:39 PM
Restricted Application removed a project: Patch-For-Review. · View Herald TranscriptMay 19 2020, 2:39 PM
Andrew triaged this task as Medium priority.May 19 2020, 4:03 PM
Andrew moved this task from Inbox to Doing on the cloud-services-team (Kanban) board.
RhinosF1 subscribed.May 20 2020, 9:39 AM
gerritbot added a comment.May 20 2020, 2:43 PM

Change 597560 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] cloudnet1003 and cloudnet1004: next install with Buster

https://gerrit.wikimedia.org/r/597560

gerritbot added a project: Patch-For-Review.May 20 2020, 2:43 PM
gerritbot added a comment.May 21 2020, 4:04 AM

Change 597560 merged by Andrew Bogott:
[operations/puppet@production] cloudnet1003 and cloudnet1004: next install with Buster

https://gerrit.wikimedia.org/r/597560

ops-monitoring-bot added a comment.May 21 2020, 12:43 PM

Script wmf-auto-reimage was launched by andrew on cumin1001.eqiad.wmnet for hosts:

cloudnet1004.eqiad.wmnet

The log can be found in /var/log/wmf-auto-reimage/202005211243_andrew_213507_cloudnet1004_eqiad_wmnet.log.

Stashbot added a comment.May 21 2020, 12:44 PM

Mentioned in SAL (#wikimedia-operations) [2020-05-21T12:44:22Z] <andrewbogott> reimaging cloudnet1004.eqiad.wmnet for T253124

ops-monitoring-bot added a comment.May 21 2020, 12:58 PM

Completed auto-reimage of hosts:

['cloudnet1004.eqiad.wmnet']

Of which those FAILED:

['cloudnet1004.eqiad.wmnet']
ops-monitoring-bot added a comment.May 21 2020, 12:59 PM

Script wmf-auto-reimage was launched by andrew on cumin1001.eqiad.wmnet for hosts:

cloudnet1004.eqiad.wmnet

The log can be found in /var/log/wmf-auto-reimage/202005211259_andrew_214571_cloudnet1004_eqiad_wmnet.log.

ops-monitoring-bot added a comment.May 21 2020, 12:59 PM

Completed auto-reimage of hosts:

['cloudnet1004.eqiad.wmnet']

Of which those FAILED:

['cloudnet1004.eqiad.wmnet']
ops-monitoring-bot added a comment.May 21 2020, 1:32 PM

Script wmf-auto-reimage was launched by andrew on cumin1001.eqiad.wmnet for hosts:

cloudnet1004.eqiad.wmnet

The log can be found in /var/log/wmf-auto-reimage/202005211332_andrew_218658_cloudnet1004_eqiad_wmnet.log.

ops-monitoring-bot added a comment.May 21 2020, 1:32 PM

Completed auto-reimage of hosts:

['cloudnet1004.eqiad.wmnet']

Of which those FAILED:

['cloudnet1004.eqiad.wmnet']
Andrew updated the task description. (Show Details)May 21 2020, 3:36 PM
ops-monitoring-bot added a comment.May 21 2020, 3:48 PM

Script wmf-auto-reimage was launched by andrew on cumin1001.eqiad.wmnet for hosts:

cloudnet1003.eqiad.wmnet

The log can be found in /var/log/wmf-auto-reimage/202005211547_andrew_239426_cloudnet1003_eqiad_wmnet.log.

Stashbot added a comment.May 21 2020, 3:48 PM

Mentioned in SAL (#wikimedia-operations) [2020-05-21T15:48:44Z] <andrewbogott> rebuilding cloudnet1003.eqiad.wmnet with Debian Buster for T253124

Andrew updated the task description. (Show Details)May 21 2020, 3:52 PM
ops-monitoring-bot added a comment.May 21 2020, 4:35 PM

Completed auto-reimage of hosts:

['cloudnet1003.eqiad.wmnet']

and were ALL successful.

Andrew closed this task as Resolved.May 21 2020, 6:41 PM
Andrew updated the task description. (Show Details)
aborrero added a subtask: T257552: cloudnet: prometheus node exporter stopped collecting nf_conntrack_entries metric.Jul 9 2020, 9:29 AM
aborrero closed subtask T257552: cloudnet: prometheus node exporter stopped collecting nf_conntrack_entries metric as Resolved.Jul 14 2020, 11:51 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL