I did some nodetool cleanups on some of the worst of the nodes, and lowered utilization by a few percent.

The RESTBase mobile-sections tables (corresponding to Cassandra keyspaces {commons,enwiki,others,wikipedia}_T_mobileoZCBVtILw5eSrwi0VIGaFVSr2jY) seem to be no longer in use (having been superseded by PCS?). See:

Combined that's almost 15TB of storage:

I propose that we TRUNCATE these tables (leaving the schema in place for now). Any objections @Jgiannelos? @akosiaris ?

I think its fine to truncate the mobile-sections related tables.

Ok, here (for posterity sake) is what I will apply:

TRUNCATE "commons_T_mobileoZCBVtILw5eSrwi0VIGaFVSr2jY".data;
TRUNCATE "enwiki_T_mobileoZCBVtILw5eSrwi0VIGaFVSr2jY".data;
TRUNCATE "others_T_mobileoZCBVtILw5eSrwi0VIGaFVSr2jY".data;
TRUNCATE "wikipedia_T_mobileoZCBVtILw5eSrwi0VIGaFVSr2jY".data;

Mentioned in SAL (#wikimedia-operations) [2024-08-28T15:52:41Z] <urandom> TRUNCATE-ing RESTBase tables ({commons,enwiki,others,wikipedia}_T_mobileoZCBVtILw5eSrwi0VIGaFVSr2jY) — T342148

In T342148#10100244, @Eevans wrote:

Ok, here (for posterity sake) is what I will apply:

TRUNCATE "commons_T_mobileoZCBVtILw5eSrwi0VIGaFVSr2jY".data;
TRUNCATE "enwiki_T_mobileoZCBVtILw5eSrwi0VIGaFVSr2jY".data;
TRUNCATE "others_T_mobileoZCBVtILw5eSrwi0VIGaFVSr2jY".data;
TRUNCATE "wikipedia_T_mobileoZCBVtILw5eSrwi0VIGaFVSr2jY".data;

That done, I'll give it a couple of hours before clearing the snapshots and actually reclaiming the space (out of an abundance of caution).

In T342148#10098113, @Eevans wrote:

The RESTBase mobile-sections tables (corresponding to Cassandra keyspaces {commons,enwiki,others,wikipedia}_T_mobileoZCBVtILw5eSrwi0VIGaFVSr2jY) seem to be no longer in use (having been superseded by PCS?). See:

Combined that's almost 15TB of storage:

I propose that we TRUNCATE these tables (leaving the schema in place for now). Any objections @Jgiannelos? @akosiaris ?

None on my side. Thanks for chasing that down!