Page MenuHomePhabricator

Ceph Data Infrastructure Request
Closed, ResolvedPublic


Request Status: New Request
Request Type: Infrastructure Request

Request Title: Ceph Data Infrastructure Request

  • Request Description:

We have identified a need for a new storage platform in order to meet the needs of teams focused on building data-centric products and features.

The platform should offer both block storage and object storage capabilities and should be suitable for use with both analytics and production oriented workloads.

As such we are planning to build an MVP of such a platform in Q2 of the 2022/2023 financial year, with the Data-Engineering team taking primary responsibility for its design and implementation. Consultation and close collaboration with the Data Persistence and Infrastructure Foundations teams will be essential in order to ensure that the deisgn meets the requirements and that the expected traffic profile is compatible with our network topology.

The end goal is to build a scalable platform that can facilitate self-service data infrastructure provision across many teams and for a wide variety of requirements. The MVP should be designed in such a way that once its value has been proven it can be promoted to a production class service without a full rebuild.

Some key use cases include

  • the ability to support Persistent Volume Claims in Kubernetes, such that we can begin to deploy stateful services on k8s
  • the ability to provide block storage to virtual machines, for enhanced flexibility in designing data processing systems on VMs
  • the ability to provide S3 and/or Swift compatible object storage as a back-end for analytics and similar workloads

In this phase we are only looking at building this platform in eqiad, although we should always consider how it would scale to a multi-DC and/or cross-DC design.

  • Indicate Priority Level: High
  • Main Requestors: Data Engineering,
  • Ideal Delivery Date: Q2 of the 2022/2023 financial year,
  • Stakeholders: Data Engineering, Machine Learning, Platform Engineering

Request Documentation

Document TypeRequired?Document/Link
Related PHAB TicketsYesT309509: Assess existing and in-development storage platforms for suitability
Product One PagerYes<add link here>
Product Requirements Document (PRD)No<add link here>
Product RoadmapNo<add link here>
Product Planning/Business CaseNo<add link here>
Product BriefNo<add link here>
Design DocYes

Event Timeline

I'm putting this link here as a reminder to myself:

This is a useful reference for running Hive on top of Ceph object storage.

BTullis added a subtask: Unknown Object (Task).Dec 7 2022, 12:08 PM
BTullis moved this task from Work In Progress to Done on the Foundational Technology Requests board.

I believe that this ticket could now be said to have been done. We requested a Ceph cluster, the hardware has been purchased, racked and the O/S installed.
I'm now tracking the installation and configuration of Ceph through T324660: Install Ceph Cluster for Data Engineering which I have added as a sub-task of this ticket.