This is a request for Ceph S3 credentials for the research team
- username: research
- resource: 4TB (arbitrary, enough to experiment with non-trivial datasets)
- use cases:
- research already uses S3/swift for shuffling data between DE infra (yarn) and ML infra (kubernetes), and will deploy workloads on the DSE cluster too. It would be preferable to use ceph for this.
- instead of battling with hdfs/kerberos, for new tools it is much easier to just use object storage (as presumably the whole DE infra will do at some point in the future). Example: datafusion-wmf already supports reading/writing from S3.