Automatic schema migrations. In a large cluster, it's been proven that automatic schema migrations do not work - the process takes a long time and needs to be closely monitored, so it has to be performed by a Cassandra DBA. Quite a bit of code and tests in restbase-mod-table-* packages is dedicated to schema migrations, and this code needs to be removed.

Just for posterity sake, I think the "...takes a long time..." aspect of this is the result of being so constrained on IO, and I expect this to be less of a pain-point after the upgrade. Even still, schema changes are an exceptional event that should be handled with great care, and efforts to programmatically manage this from the application have time and again proven brittle at best (and this experience is matched by those outside the WMF as well).

TL;DR +1

Changing the storage groups semantics The concept of storage groups allows to partition the storage by domain. Currently, it's implemented using different keyspaces per storage group. In Cassandra application-level sharding is not strictly required, however sometimes it comes handy during transitions allowing to drop data per shard for example. I'm not entirely sure if we would need to support storage groups at all since they come with a price...

So, speaking about implementation here abstractly: I can't really recall any time that the storage group concept proved to be indispensable (wrt storage, at least); The problem here is that the sharding produces extremely skewed distribution (i.e. almost everything we store is English Wikipedia).

...but if we are, we need to convert them to separate tables under one keyspace, not multiple keyspaces.

Again, speaking to implementation abstractly: I agree, this should be (should have been) organized as tables under a single keyspace.

Now, speaking concretely: If we're intent on moving away from this entirely, in other words, if post-split this is all deprecated pending replacement, is it worth a disruptive migration?

If we decide to drop support of the storage groups, we might as well drop a separate implicit _domain partitioning key.

Can we? Don't we still need the domain to guarantee uniqueness for page titles that are identical across domain/project?

Stop producing cryptic keyspace names The keyspace names have a length limit in Cassandra, so to avoid hitting the limit we were creating cryptic KS names. Dropping the storage groups mapping to KS names might allow us to finally get rid of cryptic names, make the clients select shorter names and through an error if the name is longer then the limit.

Again, I fully agree in principle, but I'm still wondering whether it makes sense to undergo such a disruptive migration if we intend to deprecate and replace all of this.

I think I have failed to describe the details of the basis of the reasoning behind this leaving a lot of room for confusion. I'll try to fix this mistake.

The goal of the next quarter is to split RESTBase into API router deployed on k8s and storage component deployed alongside Cassandra. In the grand scheme of things, the storage component will be eventually removed and replaced by individual services directly managing their own data either via Cassandra driver directly, via some slim library (the simplified restbase-mod-table-*) or through Kask, depending on the service and its needs. As a part of the split, we also want to simplify storage for certain use-cases (T215960 and T215956). This means, that the majority of storage will have to undergo migration on Cassandra level - we can't just replace the software while using the same underlying tables.

Since we will need to do storage migration anyway, w want the resulting table layout to be as close to the 'ultimate perfection' as possible, thus this ticket for discussion of which features we want to deprecate/change in our storage access libraries. Ideally, the underlying storage model should be as close to what Kask will use as possible, so we would be able to seamlessly replace accessing Cassandra via restbase-mod-table-* with accessing through Kask if the need comes.

So, speaking about implementation here abstractly: I can't really recall any time that the storage group concept proved to be indispensable

It was nice when we needed to drop data from certain keyspaces. With skewed distribution, we can fight by having the same partitioning as MySQL sharding - the shards are designed to have roughly equal distribution. However, given that now we have improved culling scripts, I don't think I see a lot of value in supporting storage groups either. Ultimately, I think I'm +1 on removing the possibility of storage groups.

If we decide to drop support of the storage groups, we might as well drop a separate implicit _domain partitioning key.

Can we? Don't we still need the domain to guarantee uniqueness for page titles that are identical across domain/project?

We can prefix the 'key' with 'domain' in the code and not have a separate implicit _domain part of the partitioning key in Cassandra schema. That will bring the schema used by the libraries closer to the schema used by Kask. However, having the implicit _domain feels kinda nice.