Page MenuHomePhabricator
Create Task
Maniphest T263677

Upgrade cloudvirt1019 and cloudvirt1020 to Buster
Closed, ResolvedPublic
Actions

Description

Cloudvirt1019 and cloudvirt1020 host:

clouddb1001.clouddb-services.eqiad1.wikimedia.cloud
clouddb1002.clouddb-services.eqiad1.wikimedia.cloud
clouddb1003.clouddb-services.eqiad1.wikimedia.cloud
clouddb1004.clouddb-services.eqiad1.wikimedia.cloud

We do not have immediate plans to move these to Ceph, which means that upgrading them will require a fair bit of downtiming/failover steps to keep toolsdb users happy.

I'm going to experiment with rebuilding hypervisors in a way that leaves their storage partition intact; in the meantime @Bstorm will figure out how to manage the downtime.

Details

SubjectRepoBranchLines +/-
Revert "cloudvirt1020: update nic names for Buster"operations/puppetproduction+3 -3
cloudvirt1020: update nic names for Busteroperations/puppetproduction+3 -3
cloudvirt1019/1020: Move to Buster on next reimageoperations/puppetproduction+0 -2
Update partman recipes for cloudvirtsoperations/puppetproduction+4 -3
Further attempt to reimage cloudvirts while preserving /srvoperations/puppetproduction+4 -54
Customize query in gerrit

Related Objects
Search...

Event Timeline

Andrew created this task.Sep 23 2020, 6:33 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSep 23 2020, 6:33 PM
Andrew added a comment.Sep 23 2020, 6:47 PM

I'd like to have this done by ~ the third week of October; by that time all the other upgrades should be unblocked.

• Bstorm added a comment.Sep 23 2020, 7:05 PM

Based on initial scribbling of plans in the subtask, I'd like to request we do cloudvirt1020 first. That way, when we failover clouddb1001 (which is on cloudvirt1019) to clouddb1002 (cloudvirt1020), it doesn't have to fail back, saving ToolsDB downtime.

Andrew added a parent task: T259399: Upgrade cloudvirts to Debian Buster.Sep 28 2020, 6:37 PM
gerritbot added a comment.Sep 29 2020, 1:54 AM

Change 630708 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Further attempt to reimage cloudvirts while preserving /srv

https://gerrit.wikimedia.org/r/630708

gerritbot added a project: Patch-For-Review.Sep 29 2020, 1:54 AM

Change 630708 merged by Andrew Bogott:
[operations/puppet@production] Further attempt to reimage cloudvirts while preserving /srv

https://gerrit.wikimedia.org/r/630708

Maintenance_bot removed a project: Patch-For-Review.Sep 29 2020, 2:10 AM
gerritbot added a comment.Sep 29 2020, 9:39 PM

Change 630954 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Update partman recipes for cloudvirts

https://gerrit.wikimedia.org/r/630954

gerritbot added a project: Patch-For-Review.Sep 29 2020, 9:39 PM
gerritbot added a comment.Sep 30 2020, 3:42 PM

Change 630954 merged by Andrew Bogott:
[operations/puppet@production] Update partman recipes for cloudvirts

https://gerrit.wikimedia.org/r/630954

Maintenance_bot removed a project: Patch-For-Review.Sep 30 2020, 4:10 PM
gerritbot added a comment.Oct 19 2020, 6:28 PM

Change 635023 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] cloudvirt1019/1020: Move to Buster on next reimage

https://gerrit.wikimedia.org/r/635023

gerritbot added a project: Patch-For-Review.Oct 19 2020, 6:28 PM
Andrew updated the task description. (Show Details)Oct 19 2020, 6:30 PM
gerritbot added a comment.Oct 20 2020, 2:22 AM

Change 635023 merged by Andrew Bogott:
[operations/puppet@production] cloudvirt1019/1020: Move to Buster on next reimage

https://gerrit.wikimedia.org/r/635023

Maintenance_bot removed a project: Patch-For-Review.Oct 20 2020, 3:10 AM
Andrew moved this task from Inbox to Doing on the cloud-services-team (Kanban) board.Oct 20 2020, 4:33 PM
Stashbot added a comment.Oct 20 2020, 5:07 PM

Mentioned in SAL (#wikimedia-cloud) [2020-10-20T17:07:27Z] <bstorm> shut down replication on clouddb1002 (now with task) T263677

Stashbot added a comment.Oct 20 2020, 5:08 PM

Mentioned in SAL (#wikimedia-cloud) [2020-10-20T17:08:13Z] <bstorm> stopping mariadb on clouddb1002 T263677

Stashbot added a comment.Oct 20 2020, 5:08 PM

Mentioned in SAL (#wikimedia-cloud) [2020-10-20T17:08:46Z] <bstorm> poweroff clouddb1002 T263677

Stashbot added a comment.Oct 20 2020, 5:13 PM

Mentioned in SAL (#wikimedia-cloud) [2020-10-20T17:13:43Z] <bstorm> stopping postgresql on clouddb1003 T263677

Stashbot added a comment.Oct 20 2020, 5:14 PM

Mentioned in SAL (#wikimedia-cloud) [2020-10-20T17:14:15Z] <bstorm> shutting down clouddb1003 T263677

gerritbot added a comment.Oct 20 2020, 5:38 PM

Change 635325 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] cloudvirt1020: update nic names for Buster

https://gerrit.wikimedia.org/r/635325

gerritbot added a project: Patch-For-Review.Oct 20 2020, 5:38 PM

Change 635325 merged by Andrew Bogott:
[operations/puppet@production] cloudvirt1020: update nic names for Buster

https://gerrit.wikimedia.org/r/635325

gerritbot added a comment.Oct 20 2020, 5:51 PM

Change 635346 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Revert "cloudvirt1020: update nic names for Buster"

https://gerrit.wikimedia.org/r/635346

gerritbot added a comment.Oct 20 2020, 5:52 PM

Change 635346 merged by Andrew Bogott:
[operations/puppet@production] Revert "cloudvirt1020: update nic names for Buster"

https://gerrit.wikimedia.org/r/635346

Maintenance_bot removed a project: Patch-For-Review.Oct 20 2020, 6:10 PM
Andrew added a comment.Oct 20 2020, 6:13 PM

cloudvirt1020 is now running Buster.

Stashbot added a comment.Oct 20 2020, 6:36 PM

Mentioned in SAL (#wikimedia-cloud) [2020-10-20T18:36:34Z] <bstorm> brought up mariadb and replication on clouddb1002 T263677

• Bstorm changed the status of subtask T263679: Plan and manage the required downtime for the clouddb-services VMs on cloudvirt1019 and cloudvirt1020 from Open to Stalled.Oct 29 2020, 12:18 AM
Andrew mentioned this in T259399: Upgrade cloudvirts to Debian Buster.Dec 8 2020, 7:05 PM
• Bstorm closed subtask T263679: Plan and manage the required downtime for the clouddb-services VMs on cloudvirt1019 and cloudvirt1020 as Resolved.Dec 17 2020, 3:09 PM
Andrew closed this task as Resolved.Dec 17 2020, 3:18 PM
Andrew claimed this task.

Cloudvirt1019 was upgraded yesterday.

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits