Page MenuHomePhabricator

Remove ar_comment from sanitarium triggers
Open, MediumPublic

Description

While the schema change arrives to the sanitarium hosts, replication breaks due to missing column ar_comment on the sanitization triggers:

Last_SQL_Error: Could not execute Write_rows_v1 event on table jawiki.archive; Unknown column 'ar_comment' in 'NEW', Error_code: 1054; handler error HA_ERR_GENERIC; the event's master log db2076-bin.001830, end_log_pos 827488672
root@cumin1001:/home/marostegui/T233135# mysql.py -hdb2095:3316 -e "show triggers like 'archive%'" jawiki
+----------------+--------+---------+------------------------------------------------+--------+---------+--------------------------+----------------+----------------------+----------------------+--------------------+
| Trigger        | Event  | Table   | Statement                                      | Timing | Created | sql_mode                 | Definer        | character_set_client | collation_connection | Database Collation |
+----------------+--------+---------+------------------------------------------------+--------+---------+--------------------------+----------------+----------------------+----------------------+--------------------+
| archive_insert | INSERT | archive | SET NEW.ar_comment = '', NEW.ar_comment_id = 0 | BEFORE | NULL    | IGNORE_BAD_TABLE_OPTIONS | root@localhost | utf8                 | utf8_general_ci      | binary             |
| archive_update | UPDATE | archive | SET NEW.ar_comment = '', NEW.ar_comment_id = 0 | BEFORE | NULL    | IGNORE_BAD_TABLE_OPTIONS | root@localhost | utf8                 | utf8_general_ci      | binary             |
+----------------+--------+---------+------------------------------------------------+--------+---------+--------------------------+----------------+----------------------+----------------------+--------------------+

The rest of the triggers should be checked just in case.

  • s1
    • codfw
    •   eqiad
  • s2
    • codfw
    •   eqiad
  • s3
    • codfw
    •   eqiad
  • s4
    • codfw
    •   eqiad
  • s5
    • codfw
    •   eqiad
  • s6
    • codfw
    •   eqiad
  • s7
    • codfw
    •   eqiad
  • s8
    • codfw
    •   eqiad
  • modules/role/files/mariadb/filtered_tables.txt

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 5 2019, 4:59 AM
Marostegui triaged this task as Medium priority.Oct 5 2019, 4:59 AM
Marostegui moved this task from Triage to In progress on the DBA board.

Mentioned in SAL (#wikimedia-operations) [2019-10-07T05:08:38Z] <marostegui> Stop replication on db2076 to modify triggers on db2096:3316 T234704

Mentioned in SAL (#wikimedia-operations) [2019-10-07T05:10:39Z] <marostegui> The above was for db2095:3316 T234704

Marostegui updated the task description. (Show Details)Oct 7 2019, 5:20 AM
Marostegui updated the task description. (Show Details)Oct 7 2019, 5:44 AM

Mentioned in SAL (#wikimedia-operations) [2019-10-10T11:35:47Z] <marostegui> Stop replication on db2077 to change triggers on db2095:3317 - T234704

Marostegui updated the task description. (Show Details)Oct 10 2019, 11:41 AM

Mentioned in SAL (#wikimedia-operations) [2019-10-16T05:11:32Z] <marostegui> Change s2 triggers for archive table from db1125:3312 T234704

Mentioned in SAL (#wikimedia-operations) [2019-10-16T05:14:55Z] <marostegui> Change s7 triggers for archive table from db1125:3317 T234704

Marostegui updated the task description. (Show Details)Oct 16 2019, 5:15 AM
This comment was removed by Marostegui.

Mentioned in SAL (#wikimedia-operations) [2019-10-16T10:17:10Z] <marostegui> Stop replication on s2 codfw master for schema change and to modify sanitarium triggers T234066 T233135 T234704

Marostegui updated the task description. (Show Details)Oct 16 2019, 10:21 AM

Mentioned in SAL (#wikimedia-operations) [2019-11-12T06:44:00Z] <marostegui> Change triggers on s5 db2094 - T234704

Marostegui updated the task description. (Show Details)Tue, Nov 12, 6:44 AM

Mentioned in SAL (#wikimedia-operations) [2019-11-14T08:42:22Z] <marostegui> Remove ar_comment from triggers on db1124:3315 - T234704

Marostegui updated the task description. (Show Details)Thu, Nov 14, 8:51 AM

Mentioned in SAL (#wikimedia-operations) [2019-11-18T09:11:49Z] <marostegui> Remove ar_comment from triggers on db2094:3318 - T234704

Marostegui updated the task description. (Show Details)Mon, Nov 18, 9:15 AM

Mentioned in SAL (#wikimedia-operations) [2019-11-25T14:38:10Z] <marostegui> Remove triggers from archive table on s1 codfw sanitarium T234704

Marostegui updated the task description. (Show Details)Mon, Nov 25, 2:39 PM

Mentioned in SAL (#wikimedia-operations) [2019-11-26T06:44:05Z] <marostegui> Remove triggers for ar_comment on db1124:3318 T234704

Marostegui updated the task description. (Show Details)Tue, Nov 26, 6:46 AM

Mentioned in SAL (#wikimedia-operations) [2019-11-28T13:31:38Z] <marostegui> Remove ar_comment triggers from db1124:3311 for enwiki.archive - T234704

Marostegui updated the task description. (Show Details)Thu, Nov 28, 1:32 PM

Mentioned in SAL (#wikimedia-operations) [2019-11-29T09:33:53Z] <marostegui> Remove triggers from db2094:3313 - T234704

Marostegui updated the task description. (Show Details)Fri, Nov 29, 9:44 AM
Marostegui updated the task description. (Show Details)Mon, Dec 2, 3:00 PM

Mentioned in SAL (#wikimedia-operations) [2019-12-03T05:47:22Z] <marostegui> Remove ar_comment triggers from s3 db1124:3313 - T234704

Marostegui updated the task description. (Show Details)Tue, Dec 3, 5:48 AM