The hard part is that on sanitarium we have filters on archive tables, so compare.py will return error

As a first approach and in order to advance some work, we can probably compare sanitarium masters with the master (or other slaves) and leave sanitarium themselves aside for now. At least we will be sure that masters and sanitarium masters are the same.
Sanitariums were originally cloned from sanitariums masters.

I started the comparison on s5 between db1070 and db1082 with

for db in $(cat mediawiki-config/dblists/s5.dblist); do ./wmfmariadbpy/wmfmariadbpy/compare.py ${db} archive ar_id db1070 db1082; done | tee s5_check.out

s5 completed, going forward

The screen name in sanitarium_check and runs on cumin1001

s6 ok, continuing with s7

for db in $(cat mediawiki-config/dblists/s7.dblist); do echo "Checking database ${db}"; ./wmfmariadbpy/wmfmariadbpy/compare.py ${db} archive ar_id db1062 db1079; done | tee s7_check.out

s7 completed, s8:

for db in $(cat mediawiki-config/dblists/s8.dblist); do echo "Checking database ${db}"; ./wmfmariadbpy/wmfmariadbpy/compare.py ${db} archive ar_id db1071 db1087; done | tee s8_check.out

s8 completed, s2:

for db in $(cat mediawiki-config/dblists/s2.dblist); do echo "Checking database ${db}"; ./wmfmariadbpy/wmfmariadbpy/compare.py ${db} archive ar_id db1066 db1074; done | tee s2_check.out

s2 done, s4:

for db in $(cat mediawiki-config/dblists/s4.dblist); do echo "Checking database ${db}"; ./wmfmariadbpy/wmfmariadbpy/compare.py ${db} archive ar_id db1068 db1121; done | tee s4_check.out

s4 done, s3:

for db in $(cat mediawiki-config/dblists/s3.dblist); do echo "Checking database ${db}"; ./wmfmariadbpy/wmfmariadbpy/compare.py ${db} archive ar_id db1075 db1077; done | tee s3_check.out

s3 finished, one difference found:

Checking database tcywiki 
Starting comparison between id 1 and 226
DIFFERENCE on db1077.eqiad.wmnet:3306: WHERE ar_id BETWEEN 1 AND 226
Execution ended, a total of 1 chunk(s) are different.

I re-run the check several times, and it was show up as an error.

In T209048#4742064, @Banyek wrote:

s3 finished, one difference found:

Checking database tcywiki 
Starting comparison between id 1 and 226
DIFFERENCE on db1077.eqiad.wmnet:3306: WHERE ar_id BETWEEN 1 AND 226
Execution ended, a total of 1 chunk(s) are different.

I re-run the check several times, and it was show up as an error.

fixed

in the meanwhile the check finished on s1 too, and found no differences

You only checked 2 servers on each comparison- you should check all of them- it takes approximately the same amount of time, speed it up with --step=100000 and automate the check with --quiet and || break 2 as the scripts returns 0 when successfully executed and other on error.

In T209048#4753191, @jcrespo wrote:

You only checked 2 servers on each comparison- you should check all of them- it takes approximately the same amount of time, speed it up with --step=100000 and automate the check with --quiet and || break 2 as the scripts returns 0 when successfully executed and other on error.

• with tcywiki I checked all the hosts after I found that diff
• I was searching the the output of the tool created with grep to find if there any other error