Background:

The Generated Data Platform (née AQS) cluster used to be treated as single tenancy. One team maintained a service (AQS) backed by a dedicated Cassandra cluster (colloquially known as The AQS Cluster™). The cluster has since been repurposed as multi-tenancy storage for any datasets of a pattern similar to that of AQS, materialized representations persisted for low-latency transactional access.

Back In The Day, this cluster had a user called aqsloader that was used for —conveniently enough— loading the AQS datasets into Cassandra, and an aqs user used by the http service (reading).

For a multi-tenant environment, the plan has been to delineate access by dataset. Think: minimizing the blast radius of an aberrant commit that resulted in data loss, or of a leaked credential. It could also be useful in applying by-dataset resource constraints, or t-shooting aberrant access patterns.

It isn't entirely clear to me how the Airflow DAGs work, but it is my understanding that loading is something that happens within the scope of each DAG (rather than there being some sort of abstraction that would limit the scope of Cassandra access to DAG implementers). Someone please let me know if I have this wrong.

Problem:

Image suggestions was provisioned with a dedicated user —image_suggestions— with SELECT and MODIFY rights, but is using the aqsloader user instead.