While the schema change arrives to the sanitarium hosts, replication breaks due to missing column ar_comment on the sanitization triggers:
Last_SQL_Error: Could not execute Write_rows_v1 event on table jawiki.archive; Unknown column 'ar_comment' in 'NEW', Error_code: 1054; handler error HA_ERR_GENERIC; the event's master log db2076-bin.001830, end_log_pos 827488672
root@cumin1001:/home/marostegui/T233135# mysql.py -hdb2095:3316 -e "show triggers like 'archive%'" jawiki +----------------+--------+---------+------------------------------------------------+--------+---------+--------------------------+----------------+----------------------+----------------------+--------------------+ | Trigger | Event | Table | Statement | Timing | Created | sql_mode | Definer | character_set_client | collation_connection | Database Collation | +----------------+--------+---------+------------------------------------------------+--------+---------+--------------------------+----------------+----------------------+----------------------+--------------------+ | archive_insert | INSERT | archive | SET NEW.ar_comment = '', NEW.ar_comment_id = 0 | BEFORE | NULL | IGNORE_BAD_TABLE_OPTIONS | root@localhost | utf8 | utf8_general_ci | binary | | archive_update | UPDATE | archive | SET NEW.ar_comment = '', NEW.ar_comment_id = 0 | BEFORE | NULL | IGNORE_BAD_TABLE_OPTIONS | root@localhost | utf8 | utf8_general_ci | binary | +----------------+--------+---------+------------------------------------------------+--------+---------+--------------------------+----------------+----------------------+----------------------+--------------------+
The rest of the triggers should be checked just in case.
- s1
- codfw
- eqiad
- s2
- codfw
- eqiad
- s3
- codfw
- eqiad
- s4
- codfw
- eqiad
- s5
- codfw
- eqiad
- s6
- codfw
- eqiad
- s7
- codfw
- eqiad
- s8
- codfw
- eqiad
- modules/role/files/mariadb/filtered_tables.txt https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/557808/