Page MenuHomePhabricator
Create Task
Maniphest T225320

Ceph Proof of Concept Build and Testing
Closed, ResolvedPublic0 Estimated Story Points
Actions

Description

This task is to start and track work on building out the Ceph proof of concept once the hardware arrives.

Deliverables will include:

  • Determine feasibility of Rook installation method
  • Try straight puppet installation method if Rook not feasible
  • A test of cloudvirt block storage connection and stress test.

[] CephFS connection and stress test from VMs Not using CephFS in phase one, only RBD images
[] RGW test for docker images (needs some parameters around it) Not using docker images

  • A capacity plan with at least some rough water marks

Details

SubjectRepoBranchLines +/-
nova-compute: set ceph nodes to use CPU features available on all cloudvirtsoperations/puppetproduction+6 -2
ceph: configure osd cluster network on bootoperations/puppetproduction+5 -1
ceph: configure jumbo frames on OSD interfacesoperations/puppetproduction+29 -0
openstack: add cloudvirt1022 back into scheduler pooloperations/puppetproduction+2 -1
Add some comments about cloudvirts that are set aside for ceph testing.operations/puppetproduction+3 -1
Customize query in gerrit

Related Objects
Search...

Event Timeline

• Bstorm triaged this task as Medium priority.Jun 7 2019, 6:33 PM
• Bstorm created this task.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 7 2019, 6:33 PM
• Bstorm added a comment.EditedJun 7 2019, 6:34 PM

For puppet installation and config of Ceph, the following packages are needed to be copied from the Ceph apt repo to use Ceph Luminous (the latest supported package available for Debian Stretch), based on local testing with that OS:
ceph
ceph-base
ceph-base-dbg
ceph-common
ceph-common-dbg
ceph-deploy
ceph-fuse
ceph-fuse-dbg
ceph-mds
ceph-mds-dbg
ceph-mgr
ceph-mgr-dbg
ceph-mon
ceph-mon-dbg
ceph-osd
ceph-osd-dbg
ceph-resource-agents
ceph-test
ceph-test-dbg
libcephfs-dev
libcephfs-java
libcephfs-jni
libcephfs2
libcephfs2-dbg
librados-dev
librados2
librados2-dbg
libradosstriper-dev
libradosstriper1
libradosstriper1-dbg
librbd-dev
librbd1
librbd1-dbg
librgw-dev
librgw2
librgw2-dbg
python-ceph
python-cephfs
python-cephfs-dbg
python-rados
python-rados-dbg
python-rbd
python-rbd-dbg
python-rgw
python-rgw-dbg
python3-ceph-argparse
python3-cephfs
python3-cephfs-dbg
python3-rados
python3-rados-dbg
python3-rbd
python3-rbd-dbg
python3-rgw
python3-rgw-dbg
rados-objclass-dev
radosgw
radosgw-dbg
rbd-fuse
rbd-fuse-dbg
rbd-mirror
rbd-mirror-dbg
rbd-nbd
rbd-nbd-dbg

• Bstorm moved this task from Inbox to Doing on the cloud-services-team (Kanban) board.Jun 7 2019, 6:35 PM
• Bstorm added a comment.EditedJun 14 2019, 11:33 PM

Note: I just confirmed locally that CephFS cannot set extended attributes (specifically the immutable attribute) in Luminous.
The feature has a tracker here: http://tracker.ceph.com/issues/10679
As this was last updated 3 years ago, I expect them to implement that as soon as one of us writes the patch basically 😛

Sage and Greg (main core devs) were quite open to the idea, but it doesn't seem to be a huge priority.

• Bstorm added a comment.Jun 14 2019, 11:51 PM

All that said, instead of CephFS, RBD can be exported as iSCSI, which, if set up with appropriate multipathing and a clusterFS (or using GaneshaNFS so that userland locking takes place), we could build an actually HA Linux NFS server with Ceph backing it in the exact same way as it would be for cloudvirts, with all the same quirks. That would mean that NFS could still do immutable bits because, well, it can do that. While there may be useful cases where CephFS makes sense, we may get much more use out of Ceph by simply using it for block devices in nearly every case. That sort of consistent use might make managing a ceph cluster "easier" as well, even if it might complicate NFS a bit (but not really much more than it already is). The topic of shipping ceph rbds to iscsi targets is not a small one, so I'm putting that away for now (but it may be worth testing locally while kicking this around).

• Bstorm updated the task description. (Show Details)Jun 14 2019, 11:51 PM
Paladox subscribed.Jun 15 2019, 12:14 AM
• JHedden subscribed.Jun 17 2019, 9:02 PM
• Bstorm updated the task description. (Show Details)Jun 24 2019, 3:30 PM
• Bstorm added a subtask: T90364: Test Ceph for instance storage.Jun 25 2019, 4:47 PM
• Bstorm added a subtask: T207590: Research CephFS as a replacement for NFS.
• Bstorm added a project: Epic.Jun 25 2019, 4:49 PM
• Bstorm moved this task from Doing to Epics on the cloud-services-team (Kanban) board.
• marilerr closed subtask T90364: Test Ceph for instance storage as Declined.Aug 24 2019, 3:23 AM
JJMC89 reopened subtask T90364: Test Ceph for instance storage as Open.Aug 24 2019, 3:23 AM
• marilerr closed subtask T90364: Test Ceph for instance storage as Declined.Aug 24 2019, 3:24 AM
JJMC89 reopened subtask T90364: Test Ceph for instance storage as Open.Aug 24 2019, 3:24 AM
• marilerr closed subtask T90364: Test Ceph for instance storage as Declined.Aug 24 2019, 3:25 AM
JJMC89 reopened subtask T90364: Test Ceph for instance storage as Open.Aug 24 2019, 3:25 AM
• marilerr closed subtask T90364: Test Ceph for instance storage as Declined.Aug 24 2019, 3:25 AM
JJMC89 reopened subtask T90364: Test Ceph for instance storage as Open.Aug 24 2019, 3:26 AM
• JHedden claimed this task.Oct 23 2019, 4:04 PM
• JHedden removed • JHedden as the assignee of this task.Oct 23 2019, 4:06 PM
• JHedden closed subtask T236819: Identify container images and packages required for rook.io and ceph as Resolved.Nov 25 2019, 3:14 PM
• JHedden closed subtask T236290: Deploy a Ceph testing environment using Rook.io on VMs as Resolved.Dec 5 2019, 3:05 PM
• JHedden added a subtask: T239917: Import Buster packages for Ceph Nautilus.Dec 5 2019, 3:08 PM
• JHedden closed subtask T240021: Reimage cloudceph mon and osd hosts as Resolved.Dec 6 2019, 10:41 PM
• JHedden closed subtask T239917: Import Buster packages for Ceph Nautilus as Resolved.Dec 10 2019, 6:11 PM
gerritbot added a comment.Dec 11 2019, 2:57 AM

Change 556308 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Add some comments about cloudvirts that are set aside for ceph testing.

https://gerrit.wikimedia.org/r/556308

gerritbot added a project: Patch-For-Review.Dec 11 2019, 2:57 AM
Andrew subscribed.Dec 11 2019, 2:58 AM

@JHedden, for hypervisor testing you can use cloudvirt1022 and/or cloudvirt2003-dev. I've added some comments to the pool hiera indicating that use.

gerritbot added a comment.Dec 11 2019, 2:59 AM

Change 556308 merged by Andrew Bogott:
[operations/puppet@production] Add some comments about cloudvirts that are set aside for ceph testing.

https://gerrit.wikimedia.org/r/556308

Maintenance_bot removed a project: Patch-For-Review.Dec 11 2019, 3:10 AM
• JHedden updated the task description. (Show Details)Dec 11 2019, 8:03 PM
• JHedden closed subtask T239918: Deploy Ceph Nautilus on Buster as Resolved.Dec 17 2019, 3:22 PM
bd808 mentioned this in T240417: Add option for cloning instances on horizon.Dec 22 2019, 6:30 PM
• JHedden closed subtask T240965: Enable private network interface on Ceph OSD and MON hosts as Resolved.Dec 23 2019, 4:03 PM
• JHedden closed subtask T240722: Fix Icingia disk space check on cloudcephosd100[1-3] servers as Resolved.Dec 23 2019, 8:52 PM
Stashbot added a comment.Jan 3 2020, 7:36 PM

Mentioned in SAL (#wikimedia-cloud) [2020-01-03T19:36:45Z] <jeh> create private flavor m1.small-ceph for testing IO limits T225320

Stashbot added a comment.Jan 13 2020, 2:47 PM

Mentioned in SAL (#wikimedia-cloud) [2020-01-13T14:47:54Z] <jeh> finished testing ceph on cloudvirt1022, revert to local disk puppet role and recreate canary instance T225320

• JHedden updated the task description. (Show Details)Jan 13 2020, 3:49 PM
gerritbot added a comment.Jan 13 2020, 3:50 PM

Change 564048 had a related patch set uploaded (by Jhedden; owner: Jhedden):
[operations/puppet@production] openstack: add cloudvirt1022 back into scheduler pool

https://gerrit.wikimedia.org/r/564048

gerritbot added a project: Patch-For-Review.Jan 13 2020, 3:50 PM

Change 564048 merged by Jhedden:
[operations/puppet@production] openstack: add cloudvirt1022 back into scheduler pool

https://gerrit.wikimedia.org/r/564048

Maintenance_bot removed a project: Patch-For-Review.Jan 13 2020, 4:11 PM
• JHedden closed subtask T240715: Configure prometheus monitoring for Ceph as Resolved.Jan 15 2020, 3:57 PM
• JHedden closed subtask T90364: Test Ceph for instance storage as Resolved.Jan 15 2020, 4:00 PM
bd808 removed a subtask: T207590: Research CephFS as a replacement for NFS.Jan 22 2020, 6:31 PM
bd808 added a parent task: T207590: Research CephFS as a replacement for NFS.
gerritbot added a comment.Feb 7 2020, 7:51 PM

Change 570947 had a related patch set uploaded (by Jhedden; owner: Jhedden):
[operations/puppet@production] ceph: configure jumbo frames on OSD interfaces

https://gerrit.wikimedia.org/r/570947

gerritbot added a project: Patch-For-Review.Feb 7 2020, 7:51 PM
gerritbot added a comment.Feb 7 2020, 8:35 PM

Change 570947 merged by Jhedden:
[operations/puppet@production] ceph: configure jumbo frames on OSD interfaces

https://gerrit.wikimedia.org/r/570947

Maintenance_bot removed a project: Patch-For-Review.Feb 7 2020, 9:10 PM
gerritbot added a comment.Feb 7 2020, 10:06 PM

Change 570954 had a related patch set uploaded (by Jhedden; owner: Jhedden):
[operations/puppet@production] ceph: configure osd cluster network on boot

https://gerrit.wikimedia.org/r/570954

gerritbot added a project: Patch-For-Review.Feb 7 2020, 10:06 PM
gerritbot added a comment.Feb 7 2020, 10:13 PM

Change 570954 merged by Jhedden:
[operations/puppet@production] ceph: configure osd cluster network on boot

https://gerrit.wikimedia.org/r/570954

Maintenance_bot removed a project: Patch-For-Review.Feb 7 2020, 11:10 PM
• JHedden closed subtask T243327: Test virtual machine migrations using Ceph based storage as Resolved.Feb 11 2020, 3:12 PM
• JHedden closed subtask T244868: Allow nova instance extra_specs to be updated on existing virtual machines as Resolved.Feb 11 2020, 10:53 PM
Stashbot added a comment.Mar 4 2020, 8:02 PM

Mentioned in SAL (#wikimedia-cloud) [2020-03-04T20:02:00Z] <jeh> add new ceph host aggregate T225320

• JHedden closed subtask T240718: Perform failover tests on Ceph storage cluster as Resolved.Mar 18 2020, 10:13 PM
gerritbot added a comment.May 15 2020, 9:27 PM

Change 596762 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] nova-compute: set ceph nodes to use CPU features available on all cloudvirts

https://gerrit.wikimedia.org/r/596762

gerritbot added a project: Patch-For-Review.May 15 2020, 9:27 PM
gerritbot added a comment.May 16 2020, 5:46 PM

Change 596762 merged by Andrew Bogott:
[operations/puppet@production] nova-compute: set ceph nodes to use CPU features available on all cloudvirts

https://gerrit.wikimedia.org/r/596762

Maintenance_bot removed a project: Patch-For-Review.May 16 2020, 6:10 PM
bd808 closed this task as Resolved.May 22 2020, 12:52 AM
bd808 assigned this task to • JHedden.
bd808 added a parent task: T194334: [Epic] Modern Cloud VPS storage layer.Oct 5 2020, 12:17 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL