Page MenuHomePhabricator

Remove ar_comment from sanitarium triggers
Open, NormalPublic

Description

While the schema change arrives to the sanitarium hosts, replication breaks due to missing column ar_comment on the sanitization triggers:

Last_SQL_Error: Could not execute Write_rows_v1 event on table jawiki.archive; Unknown column 'ar_comment' in 'NEW', Error_code: 1054; handler error HA_ERR_GENERIC; the event's master log db2076-bin.001830, end_log_pos 827488672
root@cumin1001:/home/marostegui/T233135# mysql.py -hdb2095:3316 -e "show triggers like 'archive%'" jawiki
+----------------+--------+---------+------------------------------------------------+--------+---------+--------------------------+----------------+----------------------+----------------------+--------------------+
| Trigger        | Event  | Table   | Statement                                      | Timing | Created | sql_mode                 | Definer        | character_set_client | collation_connection | Database Collation |
+----------------+--------+---------+------------------------------------------------+--------+---------+--------------------------+----------------+----------------------+----------------------+--------------------+
| archive_insert | INSERT | archive | SET NEW.ar_comment = '', NEW.ar_comment_id = 0 | BEFORE | NULL    | IGNORE_BAD_TABLE_OPTIONS | root@localhost | utf8                 | utf8_general_ci      | binary             |
| archive_update | UPDATE | archive | SET NEW.ar_comment = '', NEW.ar_comment_id = 0 | BEFORE | NULL    | IGNORE_BAD_TABLE_OPTIONS | root@localhost | utf8                 | utf8_general_ci      | binary             |
+----------------+--------+---------+------------------------------------------------+--------+---------+--------------------------+----------------+----------------------+----------------------+--------------------+

The rest of the triggers should be checked just in case.

  • s1
    • codfw
    •   eqiad
  • s2
    • codfw
    •   eqiad
  • s3
    • codfw
    •   eqiad
  • s4
    • codfw
    •   eqiad
  • s5
    • codfw
    •   eqiad
  • s6
    • codfw
    •   eqiad
  • s7
    • codfw
    •   eqiad
  • s8
    • codfw
    •   eqiad
  • modules/role/files/mariadb/filtered_tables.txt

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSat, Oct 5, 4:59 AM
Marostegui triaged this task as Normal priority.Sat, Oct 5, 4:59 AM
Marostegui moved this task from Triage to In progress on the DBA board.

Mentioned in SAL (#wikimedia-operations) [2019-10-07T05:08:38Z] <marostegui> Stop replication on db2076 to modify triggers on db2096:3316 T234704

Mentioned in SAL (#wikimedia-operations) [2019-10-07T05:10:39Z] <marostegui> The above was for db2095:3316 T234704

Marostegui updated the task description. (Show Details)Mon, Oct 7, 5:20 AM
Marostegui updated the task description. (Show Details)Mon, Oct 7, 5:44 AM

Mentioned in SAL (#wikimedia-operations) [2019-10-10T11:35:47Z] <marostegui> Stop replication on db2077 to change triggers on db2095:3317 - T234704

Marostegui updated the task description. (Show Details)Thu, Oct 10, 11:41 AM

Mentioned in SAL (#wikimedia-operations) [2019-10-16T05:11:32Z] <marostegui> Change s2 triggers for archive table from db1125:3312 T234704

Mentioned in SAL (#wikimedia-operations) [2019-10-16T05:14:55Z] <marostegui> Change s7 triggers for archive table from db1125:3317 T234704

Marostegui updated the task description. (Show Details)Wed, Oct 16, 5:15 AM
This comment was removed by Marostegui.

Mentioned in SAL (#wikimedia-operations) [2019-10-16T10:17:10Z] <marostegui> Stop replication on s2 codfw master for schema change and to modify sanitarium triggers T234066 T233135 T234704

Marostegui updated the task description. (Show Details)Wed, Oct 16, 10:21 AM