Change Details

**Request Status:** New Request **Request Type:** Infrastructure Request **Request Title:** //Ceph Data Infrastructure Request// -- - **Request Description:** We have identified a need for a new storage platform in order to meet the needs of teams focused on building data-centric products and features. The platform should offer both //block storage// and //object storage// capabilities and should be suitable for use with both //analytics// and //production// oriented workloads. As such we are planning to build an MVP of such a platform in Q2 of the 2022/2023 financial year, with the #data-engineering team taking primary responsibility for its design and implementation. Consultation and close collaboration with the //Data Persistence// and //Infrastructure Foundations// teams will be essential in order to ensure that the deisgn meets the requirements and that the expected traffic profile is compatible with our network topology. The end goal is to build a scalable platform that can facilitate self-service data infrastructure provision across many teams and for a wide variety of requirements. The MVP should be designed in such a way that once its value has been proven it can be promoted to a //production// class service without a full rebuild. Some key use cases include * the ability to support //Persistent Volume Claims// in Kubernetes, such that we can begin to deploy stateful services on k8s * the ability to provide block storage to virtual machines, for enhanced flexibility in designing data processing systems on VMs * the ability to provide `S3` and/or Swift compatible object storage as a back-end for analytics and similar workloads In this phase we are only looking at building this platform in eqiad, although we should always consider how it would scale to a multi-DC and/or cross-DC design. - **Indicate Priority Level:** High - **Main Requestors:** Data Engineering, - **Ideal Delivery Date:** Q2 of the 2022/2023 financial year, - **Stakeholders:** Data Engineering, Machine Learning, Platform Engineering **Request Documentation** -- |**Document Type**| **Required?** | **Document/Link** | | **Related PHAB Tickets** | Yes | <https://phabricator.wikimedia.org/T309509> | | **Product One Pager** | Yes |<add link here> | | **Product Requirements Document (PRD)** | No | <add link here> | | **Product Roadmap** | No | <add link here> | | **Product Planning/Business Case** | No |<add link here> | | **Product Brief** | No | <add link here> | | **Design Doc** | Yes |<https://docs.google.com/document/d/1dhAlABcM08zMcw9u01qwukhnw2bf6jQ9rKsRkuRRjdQ/edit#heading=h.xup1vq28kzqd> |

**Request Status:** New Request **Request Type:** Infrastructure Request **Request Title:** //Ceph Data Infrastructure Request// -- - **Request Description:** We have identified a need for a new storage platform in order to meet the needs of teams focused on building data-centric products and features. The platform should offer both //block storage// and //object storage// capabilities and should be suitable for use with both //analytics// and //production// oriented workloads. As such we are planning to build an MVP of such a platform in Q2 of the 2022/2023 financial year, with the #data-engineering team taking primary responsibility for its design and implementation. Consultation and close collaboration with the //Data Persistence// and //Infrastructure Foundations// teams will be essential in order to ensure that the deisgn meets the requirements and that the expected traffic profile is compatible with our network topology. The end goal is to build a scalable platform that can facilitate self-service data infrastructure provision across many teams and for a wide variety of requirements. The MVP should be designed in such a way that once its value has been proven it can be promoted to a //production// class service without a full rebuild. Some key use cases include * the ability to support //Persistent Volume Claims// in Kubernetes, such that we can begin to deploy stateful services on k8s * the ability to provide block storage to virtual machines, for enhanced flexibility in designing data processing systems on VMs * the ability to provide `S3` and/or Swift compatible object storage as a back-end for analytics and similar workloads In this phase we are only looking at building this platform in eqiad, although we should always consider how it would scale to a multi-DC and/or cross-DC design. - **Indicate Priority Level:** High - **Main Requestors:** Data Engineering, - **Ideal Delivery Date:** Q2 of the 2022/2023 financial year, - **Stakeholders:** Data Engineering, Machine Learning, Platform Engineering **Request Documentation** -- |**Document Type**| **Required?** | **Document/Link** | | **Related PHAB Tickets** | Yes | {T309509} | | **Product One Pager** | Yes |<add link here> | | **Product Requirements Document (PRD)** | No | <add link here> | | **Product Roadmap** | No | <add link here> | | **Product Planning/Business Case** | No |<add link here> | | **Product Brief** | No | <add link here> | | **Design Doc** | Yes |<https://docs.google.com/document/d/1dhAlABcM08zMcw9u01qwukhnw2bf6jQ9rKsRkuRRjdQ/edit#heading=h.xup1vq28kzqd> |

**Request Status:** New Request **Request Type:** Infrastructure Request **Request Title:** //Ceph Data Infrastructure Request// -- - **Request Description:** We have identified a need for a new storage platform in order to meet the needs of teams focused on building data-centric products and features. The platform should offer both //block storage// and //object storage// capabilities and should be suitable for use with both //analytics// and //production// oriented workloads. As such we are planning to build an MVP of such a platform in Q2 of the 2022/2023 financial year, with the #data-engineering team taking primary responsibility for its design and implementation. Consultation and close collaboration with the //Data Persistence// and //Infrastructure Foundations// teams will be essential in order to ensure that the deisgn meets the requirements and that the expected traffic profile is compatible with our network topology. The end goal is to build a scalable platform that can facilitate self-service data infrastructure provision across many teams and for a wide variety of requirements. The MVP should be designed in such a way that once its value has been proven it can be promoted to a //production// class service without a full rebuild. Some key use cases include * the ability to support //Persistent Volume Claims// in Kubernetes, such that we can begin to deploy stateful services on k8s * the ability to provide block storage to virtual machines, for enhanced flexibility in designing data processing systems on VMs * the ability to provide `S3` and/or Swift compatible object storage as a back-end for analytics and similar workloads In this phase we are only looking at building this platform in eqiad, although we should always consider how it would scale to a multi-DC and/or cross-DC design. - **Indicate Priority Level:** High - **Main Requestors:** Data Engineering, - **Ideal Delivery Date:** Q2 of the 2022/2023 financial year, - **Stakeholders:** Data Engineering, Machine Learning, Platform Engineering **Request Documentation** -- |**Document Type**| **Required?** | **Document/Link** | | **Related PHAB Tickets** | Yes | <https://phabricator.wikimedia.org/{T309509> |} | | **Product One Pager** | Yes |<add link here> | | **Product Requirements Document (PRD)** | No | <add link here> | | **Product Roadmap** | No | <add link here> | | **Product Planning/Business Case** | No |<add link here> | | **Product Brief** | No | <add link here> | | **Design Doc** | Yes |<https://docs.google.com/document/d/1dhAlABcM08zMcw9u01qwukhnw2bf6jQ9rKsRkuRRjdQ/edit#heading=h.xup1vq28kzqd> |