Nothing on HW logs that could indicate something is wrong with storage or similar.

Change 461098 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/mediawiki-config@master] mariadb: Depool db2041 to recover dbstore2002 s2

https://gerrit.wikimedia.org/r/461098

Change 461098 merged by jenkins-bot:
[operations/mediawiki-config@master] mariadb: Depool db2041 to recover dbstore2002 s2

https://gerrit.wikimedia.org/r/461098

Cloning is ongoing from db2041 to dbstore2002 /srv/sqldata, will rename later to /srv/sqldata.s2 and restart replication there quickly. After the backups run, some compression process will have to be rerun, which may take quite a lot on the dbstore host with so much load.

This oneliner will comress the tables from the direction of the smallest towards the largest:

mysql -BN -S /run/mysqld/mysqld.s2.sock -e "SELECT table_schema, table_name FROM information_Schema.tables WHERE engine='INNODB' and row_format <> 'COMPRESSED' ORDER BY DATA_LENGTH ASC" | while read db table; do mysql --skip-ssl --socket /run/mysqld/mysqld.s2.sock -e "ALTER TABLE $db.$table ROW_FORMAT=COMPRESSED;"; done

It shall be run inside a tmux/screen

In T204593#4597992, @Banyek wrote:

This oneliner will comress the tables from the direction of the smallest towards the largest:

mysql -BN -S /run/mysqld/mysqld.s2.sock -e "SELECT table_schema, table_name FROM information_Schema.tables WHERE engine='INNODB' and row_format <> 'COMPRESSED' ORDER BY DATA_LENGTH ASC" | while read db table; do mysql --skip-ssl --socket /run/mysqld/mysqld.s2.sock -e "ALTER TABLE $db.$table ROW_FORMAT=COMPRESSED;"; done

It shall be run inside a tmux/screen

That looks good to me, but you might want to add: set session sql_log_bin=0 - I tend to use it to avoid dealing with GTID and related things, just in case, as we don't trust MariaDB GTID much :-)

and stop replication beforehand!

mysql -BN -S /run/mysqld/mysqld.s2.sock -e "STOP SLAVE";
mysql -BN -S /run/mysqld/mysqld.s2.sock -e "SELECT table_schema, table_name FROM information_Schema.tables WHERE engine='INNODB' and row_format <> 'COMPRESSED' ORDER BY DATA_LENGTH ASC" | while read db table; do mysql --skip-ssl --socket /run/mysqld/mysqld.s2.sock -e "SET SESSION sql_log_bin=0; ALTER TABLE $db.$table ROW_FORMAT=COMPRESSED, FORCE"; done
mysql -BN -S /run/mysqld/mysqld.s2.sock -e "START SLAVE";

The tables are being compressed

Looking good so far: https://grafana.wikimedia.org/dashboard/db/prometheus-machine-stats?panelId=18&fullscreen&orgId=1&var-server=dbstore2002&var-datasource=codfw%20prometheus%2Fops&from=now-3h&to=now-1m

A quick not before I forgot: before the compression begun, we had 470G free space on that host.

The compression stopped and the replication was resumed.
The compression will continue after the tomorrow's backup is done.

The disk performs pretty slow, and the reason might be T205257
Until replication catch up I enabled the write cache with

hpssacli ctrl slot=0 modify dwc=enable

In T204593#4610981, @jcrespo wrote:

This certainly now have a lower priority as most of the compression needed to not run out of space was solved and the instance was rebuilt. You can also close it as resolved and create a lower priority ticket to review missing compression on dbstore200X tables.

That is already done! :-) T204930

sync_binlog was already disabled, and I don't found innodb_flatc

In T204593#4610981, @jcrespo wrote:

You can also close it as resolved and create a lower priority ticket to review missing compression on dbstore200X tables.

+1 to close this as resolved and follow up on T204930