The Data Platform Ceph clusters were expressly designed to support high-throughput analytics and general-purpose workloads using its S3 compatible object store interface.

However, whilst the two clusters are fully functional and in production use, the current use of S3 services is using a rudimentary security model where each use case is mapped to a specific Rados Gateway user, administered by hand with the radosgw-admin utility.

In order to be able to support a more diverse set of use-cases, we will need to define a more flexible security model that can enable role-based access control and other features that are already implemented for our existing Data Lake files.

The Hadoop security model for HDFS is built on Kerberos and membership of various POSIX groups, such as analytics-privatedata-users.
This model has served us well, but we will need to modify and build upon it to support more cloud-native workflows using S3.