Page MenuHomePhabricator
Create Task
Maniphest T253365

Complete build out of Ceph cluster and attach "diskless" cloudvirts
Closed, ResolvedPublic
Actions

Description

Complete the pilot project of hypervisors with Ceph instance storage:

  • Rack OSD nodes (DCOps)
  • Provision nodes as OSDs
  • Join nodes to existing cluster
  • Assign proper OpenStack roles to cloudvirt10[31-39] to use ceph storage
  • Drain and decommission cloudvirt100[1-9]
  • Profit!!

Details

Related Changes in Gerrit:
SubjectAuthorRepoBranchLines +/-
wmcs eqiad1: change ceph nodes to use the 'eqiad1-compute' ceph poolAndrew Bogottoperations/puppetproduction+1 -1
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolvedtaaviT211393 openstack-browser and horizon: Security group and floating IP quota information being pulled from Nova instead of Neutron for eqiad1-r
 ResolvedAndrewT211777 Can't get quota information from Neutron API
 ResolvedAndrewT261137 upgrade cloud-vps openstack to Openstack version 'Victoria'
 ResolveddcaroT261136 upgrade cloud-vps openstack to Openstack version 'Ussuri'
 ResolvedAndrewT261138 Upgrade Horizon to latest OpenStack release
 ResolvedAndrewT261135 upgrade cloud-vps openstack to Openstack version 'Train'
 ResolvedAndrewT261134 upgrade cloud-vps openstack to Openstack version 'Stein'
 ResolvedAndrewT259399 Upgrade cloudvirts to Debian Buster
Unknown Object (Task)
 ResolveddcaroT216195 Move cloudvirt hosts to 10Gb ethernet
 ResolvedAndrewT194334 [Epic] Modern Cloud VPS storage layer
 ResolvedAndrewT261132 Move all cloud-vps VMs to Ceph
 ResolvedAndrewT253365 Complete build out of Ceph cluster and attach "diskless" cloudvirts
 Resolved• CmjohnsonT251627 (Need By: 2020-06-20) rack/setup/install cloudvirt10[31-39]eqiad.wmnet
 ResolvedJclark-ctrT276922 cloudvirt1038: PCIe error
 ResolvedAndrewT258619 Decide on network setup for WMCS ceph cluster
 ResolvedAndrewT258826 Move cloudcephmon1001, 1002 and 1003 to the cloud-hosts1-b-eqiad vlan
 ResolvedAndrewT258968 Migrate existing ceph workload off of cloudcephosd100[1-3] and rebuild with new vlan config
 ResolvedAndrewT259057 Move cloudcephosd100[1-3] to new vlans
 ResolvedAndrewT259192 Investigate/prototype ceph backup options
 ResolveddcaroT260692 Ceph VM image backups
 Resolved• aborreroT260941 Practice restoring ceph backups
 DeclineddcaroT267195 CloudVPS: improve VM backups to make them redundant and discoverable
 ResolveddcaroT271094 [dumps] Review backup strategy
 OpenNoneT273720 [ceph][rbd] Periodically cleanup dangling snapshots
 OpenNoneT273723 [backups] Periodically cleanup non-handled backups
Unknown Object (Task)
 Resolved• CmjohnsonT251619 (Need By: 2020-06-20) rack/setup/install cloudcephosd10[04-15].wikimedia.org
 ResolvedJclark-ctrT251632 (Need By: 2020-06-12) rack/setup/install WMCS 10G switches
 Resolved• ayounsiT258764 Error: 'cloudsw1-c8-eqiad.mgmt.eqiad.wmnet' is not a valid parent for host 'cloudcephosd1004'
 Resolved• aborreroT266749 cloudcephosd1006: down, possible memory error

Event Timeline

bd808 created this task.May 22 2020, 12:57 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 22 2020, 12:57 AM
bd808 triaged this task as High priority.May 22 2020, 12:59 AM
bd808 added subtasks: T251619: (Need By: 2020-06-20) rack/setup/install cloudcephosd10[04-15].wikimedia.org, T251627: (Need By: 2020-06-20) rack/setup/install cloudvirt10[31-39]eqiad.wmnet, T251632: (Need By: 2020-06-12) rack/setup/install WMCS 10G switches.
bd808 moved this task from Inbox to Doing on the cloud-services-team (Kanban) board.
bd808 renamed this task from Complete build out of Ceph cluster to Complete build out of Ceph cluster and attach "diskless" cloudvirts.May 22 2020, 1:06 AM
bd808 updated the task description. (Show Details)
Andrew closed subtask T258619: Decide on network setup for WMCS ceph cluster as Resolved.Jul 23 2020, 5:59 PM
Andrew closed subtask T258826: Move cloudcephmon1001, 1002 and 1003 to the cloud-hosts1-b-eqiad vlan as Resolved.Jul 27 2020, 6:16 PM
Andrew reopened subtask T258826: Move cloudcephmon1001, 1002 and 1003 to the cloud-hosts1-b-eqiad vlan as Open.Jul 28 2020, 12:45 AM
Andrew closed subtask T258826: Move cloudcephmon1001, 1002 and 1003 to the cloud-hosts1-b-eqiad vlan as Resolved.Jul 28 2020, 6:39 PM
gerritbot added a comment.Jul 28 2020, 7:41 PM

Change 616897 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] wmcs eqiad1: change ceph nodes to use the 'eqiad1-compute' ceph pool

https://gerrit.wikimedia.org/r/616897

gerritbot added a project: Patch-For-Review.Jul 28 2020, 7:41 PM

Change 616897 merged by Andrew Bogott:
[operations/puppet@production] wmcs eqiad1: change ceph nodes to use the 'eqiad1-compute' ceph pool

https://gerrit.wikimedia.org/r/616897

Maintenance_bot removed a project: Patch-For-Review.Jul 28 2020, 8:10 PM
Andrew closed subtask T259057: Move cloudcephosd100[1-3] to new vlans as Resolved.Jul 28 2020, 8:56 PM
• Cmjohnson closed subtask T251619: (Need By: 2020-06-20) rack/setup/install cloudcephosd10[04-15].wikimedia.org as Resolved.Jul 29 2020, 3:25 PM
Andrew closed subtask T258968: Migrate existing ceph workload off of cloudcephosd100[1-3] and rebuild with new vlan config as Resolved.Jul 30 2020, 3:04 PM
Andrew closed subtask T251627: (Need By: 2020-06-20) rack/setup/install cloudvirt10[31-39]eqiad.wmnet as Resolved.Aug 3 2020, 5:49 PM
Andrew closed subtask T259192: Investigate/prototype ceph backup options as Resolved.Aug 19 2020, 4:26 AM
hashar mentioned this in T260916: Move CI instances to use ceph in WMCS.Aug 20 2020, 3:37 PM
Andrew added a parent task: T261132: Move all cloud-vps VMs to Ceph.Aug 24 2020, 3:05 PM
Andrew closed this task as Resolved.Aug 24 2020, 5:35 PM

The original description here included draining cloudvirt1001-1009 but I've added that under T261132. All the new hardware referred to in this task is now up and running and host VMs.

Andrew reopened subtask T259057: Move cloudcephosd100[1-3] to new vlans as Open.Aug 31 2020, 8:32 PM
Andrew closed subtask T259057: Move cloudcephosd100[1-3] to new vlans as Resolved.Sep 1 2020, 1:37 PM
Andrew closed subtask T260692: Ceph VM image backups as Resolved.Sep 7 2020, 4:56 PM
• Cmjohnson closed subtask T251632: (Need By: 2020-06-12) rack/setup/install WMCS 10G switches as Resolved.Oct 1 2020, 6:27 PM
Andrew reopened subtask T260692: Ceph VM image backups as Open.Oct 8 2020, 4:44 PM
dcaro closed subtask T260692: Ceph VM image backups as Resolved.Jul 28 2021, 1:48 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits