Page MenuHomePhabricator
Create Task
Maniphest T194334

[Epic] Modern Cloud VPS storage layer
Closed, ResolvedPublic
Actions

Description

The Cloud VPS project currently uses local storage on the labvirt hosts for virtual machine runtime images and an NFS cluster for off-VM persistent storage. This model requires that each labvirt (execution node) in the OpenStack cluster have locally attached disk sized to match the RAM and CPU of the host. It also means that the uptime of customer VMs are directly tied to the uptime of the underlying exec node. This complicates system maintenance, limits our ability to fairly distribute resources, and introduces a single point of failure for each and every VM.

  • roughtly estimated 6-9 month project
  • initial: 1.5 root, 100K hardware investment
  • sustaining: .5 root, 20K hardware investment

Related Objects
Search...

StatusSubtypeAssignedTask
Unknown Object (Task)
 ResolvedAndrewT194334 [Epic] Modern Cloud VPS storage layer
 ResolvedAndrewT261132 Move all cloud-vps VMs to Ceph
 ResolvedAndrewT253365 Complete build out of Ceph cluster and attach "diskless" cloudvirts
 Resolved• CmjohnsonT251627 (Need By: 2020-06-20) rack/setup/install cloudvirt10[31-39]eqiad.wmnet
 ResolvedJclark-ctrT276922 cloudvirt1038: PCIe error
 ResolvedAndrewT258619 Decide on network setup for WMCS ceph cluster
 ResolvedAndrewT258826 Move cloudcephmon1001, 1002 and 1003 to the cloud-hosts1-b-eqiad vlan
 ResolvedAndrewT258968 Migrate existing ceph workload off of cloudcephosd100[1-3] and rebuild with new vlan config
 ResolvedAndrewT259057 Move cloudcephosd100[1-3] to new vlans
 ResolvedAndrewT259192 Investigate/prototype ceph backup options
 ResolveddcaroT260692 Ceph VM image backups
 ResolvedaborreroT260941 Practice restoring ceph backups
 DeclineddcaroT267195 CloudVPS: improve VM backups to make them redundant and discoverable
 ResolveddcaroT271094 [dumps] Review backup strategy
 OpenNoneT273720 [ceph][rbd] Periodically cleanup dangling snapshots
 OpenNoneT273723 [backups] Periodically cleanup non-handled backups
 ResolvedAndrewT261252 Create system for migrating single VMs to ceph
 ResolvedAndrewT264380 Rebuild tools-prometheus-04
 ResolvedAndrewT270305 Ceph performance tuning
 ResolvedAndrewT263461 Openstack Glance: add ceph backend
 ResolvedAndrewT265843 Create backup service for glance images
 ResolvedAndrewT260916 Move CI instances to use ceph in WMCS
 ResolvedhasharT260930 jenkins-deploy user is not in the docker group
Unknown Object (Task)
 Resolved• CmjohnsonT251619 (Need By: 2020-06-20) rack/setup/install cloudcephosd10[04-15].wikimedia.org
 ResolvedJclark-ctrT251632 (Need By: 2020-06-12) rack/setup/install WMCS 10G switches
 ResolvedayounsiT258764 Error: 'cloudsw1-c8-eqiad.mgmt.eqiad.wmnet' is not a valid parent for host 'cloudcephosd1004'
 ResolvedaborreroT266749 cloudcephosd1006: down, possible memory error
Unknown Object (Task)
Unknown Object (Task)
 ResolvedPapaulT250846 (Need By: TBD) rack/setup/install cloudceph200[123]-dev
 Resolved• JHeddenT225320 Ceph Proof of Concept Build and Testing
 Resolved• JHeddenT90364 Test Ceph for instance storage
Unknown Object (Task)
 Resolved• JHeddenT224188 rack/setup/install (3) new osd ceph nodes
 Resolved• JHeddenT236290 Deploy a Ceph testing environment using Rook.io on VMs
 Resolved• JHeddenT236819 Identify container images and packages required for rook.io and ceph
 Resolved• JHeddenT239918 Deploy Ceph Nautilus on Buster
 Resolved• JHeddenT239917 Import Buster packages for Ceph Nautilus
 Resolved• JHeddenT240021 Reimage cloudceph mon and osd hosts
 Resolved• JHeddenT240715 Configure prometheus monitoring for Ceph
 Resolved• JHeddenT240718 Perform failover tests on Ceph storage cluster
 Resolved• JHeddenT240722 Fix Icingia disk space check on cloudcephosd100[1-3] servers
 Resolved• JHeddenT240965 Enable private network interface on Ceph OSD and MON hosts
 Resolved• JHeddenT243327 Test virtual machine migrations using Ceph based storage
 Resolved• JHeddenT244868 Allow nova instance extra_specs to be updated on existing virtual machines

Event Timeline

Harej created this task.May 10 2018, 12:05 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 10 2018, 12:05 AM
• Vvjjkkii renamed this task from [Epic] Modern Cloud VPS storage layer to 58caaaaaaa.Jul 1 2018, 1:10 AM
• Vvjjkkii triaged this task as High priority.
• Vvjjkkii added projects: CheckUser, Connected-Open-Heritage-Batch-uploads (RAÄ-KMB_1_2017-02), Tamil-Sites, Gamepress, Hashtags, Jade, KartoEditor, Language-2018-Apr-June, New-Editor-Experiences, Mail, TCB-Team (now WMDE-TechWish).
• Vvjjkkii updated the task description. (Show Details)
• Vvjjkkii removed a subscriber: Aklapper.
CommunityTechBot renamed this task from 58caaaaaaa to [Epic] Modern Cloud VPS storage layer.Jul 1 2018, 2:32 PM
CommunityTechBot raised the priority of this task from High to Needs Triage.
CommunityTechBot updated the task description. (Show Details)
CommunityTechBot removed projects: TCB-Team (now WMDE-TechWish), Mail, New-Editor-Experiences, Language-2018-Apr-June, KartoEditor, Jade, Hashtags, Gamepress, Tamil-Sites, Connected-Open-Heritage-Batch-uploads (RAÄ-KMB_1_2017-02), CheckUser.
CommunityTechBot added a subscriber: Aklapper.
• GTirloni triaged this task as High priority.Mar 21 2019, 8:10 PM
• GTirloni added a project: cloud-services-team (Kanban).
bd808 moved this task from Inbox to Epics on the cloud-services-team (Kanban) board.Jun 16 2019, 4:31 PM
bd808 edited projects, added cloud-services-team (FY2019-20); removed cloud-services-team (Kanban).Jul 2 2019, 9:38 PM
bd808 edited projects, added cloud-services-team (Kanban); removed cloud-services-team (FY2019-20).Jan 9 2020, 10:07 PM
bd808 added subtasks: T261132: Move all cloud-vps VMs to Ceph, T263461: Openstack Glance: add ceph backend, T260916: Move CI instances to use ceph in WMCS, Unknown Object (Task), Unknown Object (Task), Unknown Object (Task), T90364: Test Ceph for instance storage, T225320: Ceph Proof of Concept Build and Testing, T224188: rack/setup/install (3) new osd ceph nodes.Oct 5 2020, 12:17 AM
Andrew closed subtask T263461: Openstack Glance: add ceph backend as Resolved.Oct 21 2020, 2:35 AM
Andrew closed subtask T260916: Move CI instances to use ceph in WMCS as Resolved.Oct 29 2020, 2:39 PM
Andrew closed subtask T261132: Move all cloud-vps VMs to Ceph as Resolved.Nov 18 2020, 7:51 PM
bd808 closed this task as Resolved.Feb 8 2022, 5:34 PM
bd808 assigned this task to Andrew.
bd808 subscribed.

Ceph storage has been implemented and is now used for nearly all Cloud VPS instances! See https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Ceph for more information on the service.

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL