Request Status: New Request
Request Type: Infrastructure Request
Request Title: K8 DSE Kubernetes Cluster
- Request Description:
This project was originally doing by the codename of the Train Wing cluster and it was proposed by the Machine-Learning-Team.
We in the Data-Engineering team have muscled in on offered to help with this project and we will be moving forward with building the cluster.
It has now been renamed the DSE cluster, so that it is clear that it is to be a natural home for Data Science, Engineering (plus Machine Learning and Analytics) workloads.
Some key use cases include
- Running a full Kubeflow stack for training ML models
- Host the Data Warehouse & Enable trusted datasets.
- Provide a place to host & register Data-Team Applications
- the ability to integrate S3 and/or Swift compatible object storage as a back-end for analytics and similar workloads
In this phase we are only looking at building this platform in eqiad, although we should always consider how it would scale to a multi-DC and/or cross-DC design.
- Indicate Priority Level: High
- Main Requestors: Data Engineering,
- Ideal Delivery Date: Q1 of the 2022/2023 financial year,
- Stakeholders: Data Engineering, Machine Learning, Platform Engineering
Request Documentation
| Document Type | Required? | Document/Link |
| Related PHAB Tickets | Yes | T310195: Ceph Data Infrastructure Request |
| Product One Pager | Yes | <add link here> |
| Product Requirements Document (PRD) | No | <add link here> |
| Product Roadmap | No | <add link here> |
| Product Planning/Business Case | No | <add link here> |
| Product Brief | No | <add link here> |
| Design Doc | Yes | Design Document - DSE K8S Cluster |