Mentioned in SAL (#wikimedia-operations) [2022-08-05T13:27:09Z] <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depool hosts with fragile power supply (T314559 T314628)', diff saved to https://phabricator.wikimedia.org/P32292 and previous config saved to /var/cache/conftool/dbconfig/20220805-132709-ladsgroup.json

@Ladsgroup @Marostegui for reference, this is the couple of one liners I am using on cumin2002 to check the latest 2 million rows for each table:

# mysql.py -BN -h es1021 -e "SELECT table_schema FROM information_schema.tables WHERE table_name = 'blobs_cluster26' ORDER BY table_schema" | while read db; do mysql.py -BN -h es1021 $db -e "SELECT '$db', max(blob_id) FROM blobs_cluster26"; done | tee tables_to_check.txt

# grep -v NULL tables_to_check.txt | while read db rows; do echo -e "\n== $db ==\n"; db-compare $db blobs_cluster26 blob_id es1021 es2021 --step=100 --from-value=$(($rows - 2000000)) || break ; done

I chose 2 million rows as that should cover way beyond 2 days of data- while not checking the full 90 million enwiki rows or the 300 million wikidata rows from the previous years.

Mentioned in SAL (#wikimedia-operations) [2022-08-09T08:24:18Z] <jynus> starting data check using es1021 and es2021, expect increased read traffic T314559

After 38 hours of checking and 7 milion rows compared to eqiad's es1021, I can confidently say that data was in a good state after the crash.

\* Sorry, there was 7 million SELECT operations, which meant between 100 million and 700 million rows