Page MenuHomePhabricator
Create Task
Maniphest T259399

Upgrade cloudvirts to Debian Buster
Closed, ResolvedPublic
Actions

Description

It ought to be possible to upgrade the ceph-enabled cloudvirts immediately, and then the others as they're drained and converted to ceph.

Details

Show related patches Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolvedtaaviT211393 openstack-browser and horizon: Security group and floating IP quota information being pulled from Nova instead of Neutron for eqiad1-r
 ResolvedAndrewT211777 Can't get quota information from Neutron API
 ResolvedAndrewT261137 upgrade cloud-vps openstack to Openstack version 'Victoria'
 ResolveddcaroT261136 upgrade cloud-vps openstack to Openstack version 'Ussuri'
 ResolvedAndrewT261138 Upgrade Horizon to latest OpenStack release
 ResolvedAndrewT261135 upgrade cloud-vps openstack to Openstack version 'Train'
 ResolvedAndrewT261134 upgrade cloud-vps openstack to Openstack version 'Stein'
 ResolvedAndrewT259399 Upgrade cloudvirts to Debian Buster
 ResolvedAndrewT263677 Upgrade cloudvirt1019 and cloudvirt1020 to Buster
 ResolvedBstormT263679 Plan and manage the required downtime for the clouddb-services VMs on cloudvirt1019 and cloudvirt1020
 ResolvedAndrewT194334 [Epic] Modern Cloud VPS storage layer
 ResolvedAndrewT261132 Move all cloud-vps VMs to Ceph
 ResolvedAndrewT253365 Complete build out of Ceph cluster and attach "diskless" cloudvirts
 ResolvedAndrewT258619 Decide on network setup for WMCS ceph cluster
 ResolvedAndrewT258826 Move cloudcephmon1001, 1002 and 1003 to the cloud-hosts1-b-eqiad vlan
 ResolvedAndrewT258968 Migrate existing ceph workload off of cloudcephosd100[1-3] and rebuild with new vlan config
 ResolvedAndrewT259057 Move cloudcephosd100[1-3] to new vlans
 ResolvedAndrewT259192 Investigate/prototype ceph backup options
 ResolveddcaroT260692 Ceph VM image backups
 ResolvedaborreroT260941 Practice restoring ceph backups
 DeclineddcaroT267195 CloudVPS: improve VM backups to make them redundant and discoverable
 ResolveddcaroT271094 [dumps] Review backup strategy
 OpenNoneT273720 [ceph][rbd] Periodically cleanup dangling snapshots
 OpenNoneT273723 [backups] Periodically cleanup non-handled backups
 ResolvedAndrewT261252 Create system for migrating single VMs to ceph
 ResolvedAndrewT264380 Rebuild tools-prometheus-04
 ResolvedAndrewT270305 Ceph performance tuning
Unknown Object (Task)
 Resolved• CmjohnsonT251619 (Need By: 2020-06-20) rack/setup/install cloudcephosd10[04-15].wikimedia.org
 ResolvedayounsiT258764 Error: 'cloudsw1-c8-eqiad.mgmt.eqiad.wmnet' is not a valid parent for host 'cloudcephosd1004'
 ResolvedaborreroT266749 cloudcephosd1006: down, possible memory error
Unknown Object (Task)
 Resolved• CmjohnsonT251627 (Need By: 2020-06-20) rack/setup/install cloudvirt10[31-39]eqiad.wmnet
 ResolvedJclark-ctrT251632 (Need By: 2020-06-12) rack/setup/install WMCS 10G switches
 ResolvedJclark-ctrT276922 cloudvirt1038: PCIe error

Event Timeline

Andrew created this task.Jul 31 2020, 10:44 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 31 2020, 10:44 PM
Andrew added a comment.Jul 31 2020, 10:45 PM

this fix is definitely needed:

https://ask.openstack.org/en/question/120060/neutron-failing-to-deploy-with-policy-drop-not-allowed-for-user-defined-chains/

gerritbot added a comment.Jul 31 2020, 11:42 PM

Change 617791 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] openstack::nova::compute::service::rocky::buster: use legacy ebtables

https://gerrit.wikimedia.org/r/617791

gerritbot added a project: Patch-For-Review.Jul 31 2020, 11:42 PM
gerritbot added a comment.Jul 31 2020, 11:45 PM

Change 617791 merged by Andrew Bogott:
[operations/puppet@production] openstack::nova::compute::service::rocky::buster: use legacy ebtables

https://gerrit.wikimedia.org/r/617791

Maintenance_bot removed a project: Patch-For-Review.Aug 1 2020, 12:10 AM
Andrew added a comment.Aug 3 2020, 5:52 PM

andrew@cumin1001:~$ sudo cumin --force --timeout 500 "cloudvirt1*" "lsb_release -sc"
37 hosts will be targeted:
cloudvirt[1001-1009,1012-1039].eqiad.wmnet
FORCE mode enabled, continuing without confirmation

NODE GROUP

(7) cloudvirt[1033-1039].eqiad.wmnet

  • OUTPUT of 'lsb_release -sc' -----

buster

NODE GROUP

(30) cloudvirt[1001-1009,1012-1032].eqiad.wmnet

  • OUTPUT of 'lsb_release -sc' -----

stretch

gerritbot added a comment.Aug 3 2020, 5:54 PM

Change 618124 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] cloudvirt1031, 1032 -> buster

https://gerrit.wikimedia.org/r/618124

gerritbot added a project: Patch-For-Review.Aug 3 2020, 5:54 PM
gerritbot added a comment.Aug 3 2020, 5:56 PM

Change 618124 merged by Andrew Bogott:
[operations/puppet@production] cloudvirt1031, 1032 -> buster

https://gerrit.wikimedia.org/r/618124

ops-monitoring-bot added a comment.Aug 3 2020, 5:58 PM

Script wmf-auto-reimage was launched by andrew on cumin1001.eqiad.wmnet for hosts:

['cloudvirt1032.eqiad.wmnet', 'cloudvirt1031.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/202008031758_andrew_23683.log.

Maintenance_bot removed a project: Patch-For-Review.Aug 3 2020, 6:10 PM
ops-monitoring-bot added a comment.Aug 3 2020, 6:28 PM

Completed auto-reimage of hosts:

['cloudvirt1032.eqiad.wmnet', 'cloudvirt1031.eqiad.wmnet']

Of which those FAILED:

['cloudvirt1032.eqiad.wmnet', 'cloudvirt1031.eqiad.wmnet']
ops-monitoring-bot added a comment.Aug 10 2020, 1:57 PM

Script wmf-auto-reimage was launched by andrew on cumin1001.eqiad.wmnet for hosts:

['cloudvirt1032.eqiad.wmnet', 'cloudvirt1031.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/202008101357_andrew_25207.log.

ops-monitoring-bot added a comment.Aug 10 2020, 2:29 PM

Completed auto-reimage of hosts:

['cloudvirt1031.eqiad.wmnet', 'cloudvirt1032.eqiad.wmnet']

Of which those FAILED:

['cloudvirt1031.eqiad.wmnet', 'cloudvirt1032.eqiad.wmnet']
gerritbot added a comment.Aug 10 2020, 3:47 PM

Change 619315 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] cloudvirt103[1,2]: update nic names for Buster

https://gerrit.wikimedia.org/r/619315

gerritbot added a project: Patch-For-Review.Aug 10 2020, 3:47 PM

Change 619315 merged by Andrew Bogott:
[operations/puppet@production] cloudvirt103[1,2]: update nic names for Buster

https://gerrit.wikimedia.org/r/619315

Maintenance_bot removed a project: Patch-For-Review.Aug 10 2020, 4:10 PM
Andrew moved this task from Inbox to Doing on the cloud-services-team (Kanban) board.Aug 11 2020, 4:04 PM
Andrew triaged this task as Medium priority.Aug 11 2020, 4:41 PM
Andrew added a subtask: T261132: Move all cloud-vps VMs to Ceph.Aug 24 2020, 3:07 PM
Andrew added a parent task: T261134: upgrade cloud-vps openstack to Openstack version 'Stein'.Aug 24 2020, 3:22 PM
aborrero subscribed.Aug 24 2020, 4:07 PM
NOTE: I tested ebtables-nft 1.8.3 from buster-bpo and it doesn't work. We would need 1.8.5 per iptables upstream changelog (among support).
Andrew moved this task from Doing to Soon! on the cloud-services-team (Kanban) board.Sep 8 2020, 4:42 PM
Andrew added a subtask: T263677: Upgrade cloudvirt1019 and cloudvirt1020 to Buster.Sep 28 2020, 6:37 PM
gerritbot added a comment.Sep 28 2020, 6:37 PM

Change 630656 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Move cloudvirt1012,13 and 14 to ceph and Buster

https://gerrit.wikimedia.org/r/630656

gerritbot added a project: Patch-For-Review.Sep 28 2020, 6:37 PM
gerritbot added a comment.Sep 28 2020, 7:56 PM

Change 630656 merged by Andrew Bogott:
[operations/puppet@production] Move cloudvirt1012,13 and 14 to ceph and Buster

https://gerrit.wikimedia.org/r/630656

Maintenance_bot removed a project: Patch-For-Review.Sep 28 2020, 8:11 PM
gerritbot added a comment.Sep 29 2020, 2:26 AM

Change 630712 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Update nic names for cloudvirt1012, 1013, 1014

https://gerrit.wikimedia.org/r/630712

gerritbot added a project: Patch-For-Review.Sep 29 2020, 2:26 AM
gerritbot added a comment.Sep 29 2020, 2:30 AM

Change 630712 merged by Andrew Bogott:
[operations/puppet@production] Update nic names for cloudvirt1012, 1013, 1014

https://gerrit.wikimedia.org/r/630712

Maintenance_bot removed a project: Patch-For-Review.Sep 29 2020, 3:10 AM
gerritbot added a comment.Sep 29 2020, 8:40 PM

Change 630941 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Cloudvirt1016 -> Buster/Ceph

https://gerrit.wikimedia.org/r/630941

gerritbot added a project: Patch-For-Review.Sep 29 2020, 8:40 PM
gerritbot added a comment.Sep 29 2020, 8:42 PM

Change 630941 merged by Andrew Bogott:
[operations/puppet@production] Cloudvirt1016 -> Buster/Ceph

https://gerrit.wikimedia.org/r/630941

gerritbot added a comment.Oct 1 2020, 3:59 AM

Change 631325 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] cloudvirt1017 to Buster and Ceph

https://gerrit.wikimedia.org/r/631325

gerritbot added a comment.Oct 1 2020, 4:01 AM

Change 631325 merged by Andrew Bogott:
[operations/puppet@production] cloudvirt1017 to Buster and Ceph

https://gerrit.wikimedia.org/r/631325

gerritbot added a comment.Oct 1 2020, 2:08 PM

Change 631450 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] cloudvirt-wdqs* to Buster

https://gerrit.wikimedia.org/r/631450

gerritbot added a comment.Oct 1 2020, 2:10 PM

Change 631450 merged by Andrew Bogott:
[operations/puppet@production] cloudvirt-wdqs* to Buster

https://gerrit.wikimedia.org/r/631450

gerritbot added a comment.Oct 1 2020, 9:22 PM

Change 631541 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] cloudvirt1018 to Buster/Ceph

https://gerrit.wikimedia.org/r/631541

gerritbot added a comment.Oct 1 2020, 9:41 PM

Change 631541 merged by Andrew Bogott:
[operations/puppet@production] cloudvirt1018 to Buster/Ceph

https://gerrit.wikimedia.org/r/631541

gerritbot added a comment.Oct 5 2020, 5:19 PM

Change 632296 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Cloudvirt1021 and 1022 to Buster and Ceph

https://gerrit.wikimedia.org/r/632296

gerritbot added a comment.Oct 5 2020, 8:09 PM

Change 632296 merged by Andrew Bogott:
[operations/puppet@production] Cloudvirt1021 and 1022 to Buster and Ceph

https://gerrit.wikimedia.org/r/632296

gerritbot added a comment.Oct 6 2020, 8:13 PM

Change 632545 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] cloudvirt1023: to Ceph and Buster

https://gerrit.wikimedia.org/r/632545

gerritbot added a comment.Oct 7 2020, 3:19 AM

Change 632545 merged by Andrew Bogott:
[operations/puppet@production] cloudvirt1023: to Ceph and Buster

https://gerrit.wikimedia.org/r/632545

gerritbot added a comment.Oct 8 2020, 5:46 PM

Change 632972 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] cloudvirt1023: update nic names

https://gerrit.wikimedia.org/r/632972

gerritbot added a comment.Oct 8 2020, 5:48 PM

Change 632972 merged by Andrew Bogott:
[operations/puppet@production] cloudvirt1023: update nic names

https://gerrit.wikimedia.org/r/632972

gerritbot added a comment.Oct 22 2020, 2:10 AM

Change 635680 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Cloudvirt1025 and 1029 => Buster

https://gerrit.wikimedia.org/r/635680

gerritbot added a comment.Oct 22 2020, 2:12 AM

Change 635680 merged by Andrew Bogott:
[operations/puppet@production] Cloudvirt1025 and 1029 => Buster

https://gerrit.wikimedia.org/r/635680

gerritbot added a comment.Oct 22 2020, 4:35 PM

Change 635858 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Remaining eqiad1 cloudvirts to Buster

https://gerrit.wikimedia.org/r/635858

gerritbot added a comment.Oct 22 2020, 4:36 PM

Change 635858 merged by Andrew Bogott:
[operations/puppet@production] Remaining eqiad1 cloudvirts to Buster

https://gerrit.wikimedia.org/r/635858

gerritbot added a comment.Oct 26 2020, 10:29 PM

Change 636507 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Cloudvirt1027/1028 to ceph and backy2

https://gerrit.wikimedia.org/r/636507

gerritbot added a comment.Oct 26 2020, 10:31 PM

Change 636507 merged by Andrew Bogott:
[operations/puppet@production] Cloudvirt1027/1028 to ceph and backy2

https://gerrit.wikimedia.org/r/636507

Andrew closed subtask T261132: Move all cloud-vps VMs to Ceph as Resolved.Nov 18 2020, 7:51 PM
gerritbot added a comment.Dec 2 2020, 11:12 PM

Change 644943 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Cloudvirt1025-1030: Update nic names for 10Gb/Buster

https://gerrit.wikimedia.org/r/644943

gerritbot added a comment.Dec 2 2020, 11:13 PM

Change 644943 merged by Andrew Bogott:
[operations/puppet@production] Cloudvirt1025-1030: Update nic names for 10Gb/Buster

https://gerrit.wikimedia.org/r/644943

Andrew added a comment.Dec 8 2020, 7:05 PM

All cloudvirts are now running Buster save cloudvirt1019, which is T263677.

Andrew closed this task as Resolved.Dec 17 2020, 3:18 PM
Andrew closed subtask T263677: Upgrade cloudvirt1019 and cloudvirt1020 to Buster as Resolved.
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL