Page MenuHomePhabricator
Create Task
Maniphest T261132

Move all cloud-vps VMs to Ceph
Closed, ResolvedPublic
Actions

Description

  • toolforge worker nodes rebuilt on or moved to ceph
  • new ceph-enabled flavors added to replace existing flavors
  • move VMs from cloudvirt1001-1009 to ceph, mark cloudvirt1001-1009 for decom
  • convert remaining cloudvirts to ceph
  • cheer

Details

SubjectRepoBranchLines +/-
Make clouddvirt1030 a ceph-enabled hypervisoroperations/puppetproduction+1 -5
wmcs: change cloudvirt1025, 26 and 29 to ceph-enabled hypervisorsoperations/puppetproduction+6 -2
WMF Hack: hide non-ceph flavors from the instance creation workflowopenstack/horizon/horizontrain+13 -0
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolvedtaaviT211393 openstack-browser and horizon: Security group and floating IP quota information being pulled from Nova instead of Neutron for eqiad1-r
 ResolvedAndrewT211777 Can't get quota information from Neutron API
 ResolvedAndrewT261137 upgrade cloud-vps openstack to Openstack version 'Victoria'
 ResolveddcaroT261136 upgrade cloud-vps openstack to Openstack version 'Ussuri'
 ResolvedAndrewT261138 Upgrade Horizon to latest OpenStack release
 ResolvedAndrewT261135 upgrade cloud-vps openstack to Openstack version 'Train'
 ResolvedAndrewT261134 upgrade cloud-vps openstack to Openstack version 'Stein'
 ResolvedAndrewT259399 Upgrade cloudvirts to Debian Buster
Unknown Object (Task)
 ResolveddcaroT216195 Move cloudvirt hosts to 10Gb ethernet
 ResolvedAndrewT194334 [Epic] Modern Cloud VPS storage layer
 ResolvedAndrewT261132 Move all cloud-vps VMs to Ceph
 ResolvedAndrewT253365 Complete build out of Ceph cluster and attach "diskless" cloudvirts
 ResolvedCmjohnsonT251627 (Need By: 2020-06-20) rack/setup/install cloudvirt10[31-39]eqiad.wmnet
 ResolvedJclark-ctrT276922 cloudvirt1038: PCIe error
 ResolvedAndrewT258619 Decide on network setup for WMCS ceph cluster
 ResolvedAndrewT258826 Move cloudcephmon1001, 1002 and 1003 to the cloud-hosts1-b-eqiad vlan
 ResolvedAndrewT258968 Migrate existing ceph workload off of cloudcephosd100[1-3] and rebuild with new vlan config
 ResolvedAndrewT259057 Move cloudcephosd100[1-3] to new vlans
 ResolvedAndrewT259192 Investigate/prototype ceph backup options
 ResolveddcaroT260692 Ceph VM image backups
 ResolvedaborreroT260941 Practice restoring ceph backups
 DeclineddcaroT267195 CloudVPS: improve VM backups to make them redundant and discoverable
 ResolveddcaroT271094 [dumps] Review backup strategy
 OpenNoneT273720 [ceph][rbd] Periodically cleanup dangling snapshots
 OpenNoneT273723 [backups] Periodically cleanup non-handled backups
 ResolvedAndrewT261252 Create system for migrating single VMs to ceph
 ResolvedAndrewT264380 Rebuild tools-prometheus-04
 ResolvedAndrewT270305 Ceph performance tuning
Unknown Object (Task)
 ResolvedCmjohnsonT251619 (Need By: 2020-06-20) rack/setup/install cloudcephosd10[04-15].wikimedia.org
 ResolvedJclark-ctrT251632 (Need By: 2020-06-12) rack/setup/install WMCS 10G switches
 ResolvedayounsiT258764 Error: 'cloudsw1-c8-eqiad.mgmt.eqiad.wmnet' is not a valid parent for host 'cloudcephosd1004'
 ResolvedaborreroT266749 cloudcephosd1006: down, possible memory error

Event Timeline

Andrew created this task.Aug 24 2020, 3:05 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 24 2020, 3:05 PM
Andrew added a subtask: T253365: Complete build out of Ceph cluster and attach "diskless" cloudvirts.Aug 24 2020, 3:05 PM
Andrew added a parent task: T216195: Move cloudvirt hosts to 10Gb ethernet.
Andrew added a parent task: T259399: Upgrade cloudvirts to Debian Buster.
Andrew closed subtask T253365: Complete build out of Ceph cluster and attach "diskless" cloudvirts as Resolved.Aug 24 2020, 5:35 PM
Andrew mentioned this in T253365: Complete build out of Ceph cluster and attach "diskless" cloudvirts.
Andrew updated the task description. (Show Details)Sep 2 2020, 6:29 PM
Andrew closed subtask T261252: Create system for migrating single VMs to ceph as Resolved.Sep 2 2020, 7:21 PM
Andrew triaged this task as High priority.Sep 8 2020, 4:15 PM
Andrew moved this task from Inbox to Soon! on the cloud-services-team (Kanban) board.
Andrew updated the task description. (Show Details)Sep 17 2020, 3:55 PM
Andrew added a comment.Sep 23 2020, 6:17 PM

A few special cases:

clouddb-wikilabels-01.clouddb-services.eqiad1.wikimedia.cloud and clouddb-wikilabels-02.clouddb-services.eqiad1.wikimedia.cloud should be moved to Ceph but they'll need special flavors -- much smaller disk than currently allocated and also probably a lifted IO throttle. If possible coordinate this with the wikilabels folks.

clouddb-100[1-4] can stay on local storage for now but their hypervisors need to be rebuild with Buster.

Bstorm added a subtask: T264380: Rebuild tools-prometheus-04.Oct 2 2020, 12:00 AM
bd808 added a parent task: T194334: [Epic] Modern Cloud VPS storage layer.Oct 5 2020, 12:17 AM
Andrew closed subtask T264380: Rebuild tools-prometheus-04 as Resolved.Oct 15 2020, 2:17 PM
gerritbot added a comment.Oct 16 2020, 4:14 PM

Change 634545 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[openstack/horizon/horizon@train] WMF Hack: hide non-ceph flavors from the instance creation workflow

https://gerrit.wikimedia.org/r/634545

gerritbot added a project: Patch-For-Review.Oct 16 2020, 4:14 PM

Change 634545 merged by Andrew Bogott:
[openstack/horizon/horizon@train] WMF Hack: hide non-ceph flavors from the instance creation workflow

https://gerrit.wikimedia.org/r/634545

Maintenance_bot removed a project: Patch-For-Review.Oct 16 2020, 5:10 PM
Bstorm mentioned this in T216422: Virtualize NFS servers used exclusively by Cloud VPS tenants.Oct 21 2020, 8:38 PM
gerritbot added a comment.Oct 22 2020, 9:11 PM

Change 635893 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] wmcs: change cloudvirt1025, 26 and 29 to ceph-enabled hypervisors

https://gerrit.wikimedia.org/r/635893

gerritbot added a project: Patch-For-Review.Oct 22 2020, 9:11 PM
gerritbot added a comment.Oct 22 2020, 9:15 PM

Change 635893 merged by Andrew Bogott:
[operations/puppet@production] wmcs: change cloudvirt1025, 26 and 29 to ceph-enabled hypervisors

https://gerrit.wikimedia.org/r/635893

Maintenance_bot removed a project: Patch-For-Review.Oct 22 2020, 10:10 PM
Andrew updated the task description. (Show Details)Oct 28 2020, 2:13 AM
Andrew updated the task description. (Show Details)
Andrew added a comment.Oct 28 2020, 2:19 AM

Other than the special-case VMs in clouddb-services and wdqs VMs (which may remain with local storage for good), all VMs are now hosted on Ceph. Cloudvirt1025-30 are awaiting network upgrades before being rebuilt as Ceph-enabled hypervisors.

Andrew closed this task as Resolved.Nov 18 2020, 7:51 PM

I'm closing this; upgrading and moving remaining hardware can be dealt with via different tasks.

gerritbot added a comment.Dec 1 2020, 9:30 PM

Change 644621 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Make clouddvirt1030 a ceph-enabled hypervisor

https://gerrit.wikimedia.org/r/644621

gerritbot added a project: Patch-For-Review.Dec 1 2020, 9:30 PM
gerritbot added a comment.Dec 2 2020, 5:15 AM

Change 644621 merged by Andrew Bogott:
[operations/puppet@production] Make clouddvirt1030 a ceph-enabled hypervisor

https://gerrit.wikimedia.org/r/644621

Maintenance_bot removed a project: Patch-For-Review.Dec 2 2020, 6:10 AM
Andrew closed subtask T270305: Ceph performance tuning as Resolved.Jan 4 2021, 5:47 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL