Page MenuHomePhabricator

Requesting Ceph S3 credentials for research
Open, Needs TriagePublic

Description

This is a request for Ceph S3 credentials for the research team

  • username: research
  • resource: 4TB (arbitrary, enough to experiment with non-trivial datasets)
  • use cases:
    • research already uses S3/swift for shuffling data between DE infra (yarn) and ML infra (kubernetes), and will deploy workloads on the DSE cluster too. It would be preferable to use ceph for this.
    • instead of battling with hdfs/kerberos, for new tools it is much easier to just use object storage (as presumably the whole DE infra will do at some point in the future). Example: datafusion-wmf already supports reading/writing from S3.

Event Timeline

bking renamed this task from Ceph S3 credentials for research to Requesting Ceph S3 credentials for research.Tue, Feb 4, 6:20 PM
bking subscribed.

Hi Fabian,

My team is at an offsite this week, but we can take a look at this once we get back.