Page MenuHomePhabricator

Remove ar_comment from sanitarium triggers
Closed, ResolvedPublic

Description

While the schema change arrives to the sanitarium hosts, replication breaks due to missing column ar_comment on the sanitization triggers:

Last_SQL_Error: Could not execute Write_rows_v1 event on table jawiki.archive; Unknown column 'ar_comment' in 'NEW', Error_code: 1054; handler error HA_ERR_GENERIC; the event's master log db2076-bin.001830, end_log_pos 827488672
root@cumin1001:/home/marostegui/T233135# mysql.py -hdb2095:3316 -e "show triggers like 'archive%'" jawiki
+----------------+--------+---------+------------------------------------------------+--------+---------+--------------------------+----------------+----------------------+----------------------+--------------------+
| Trigger        | Event  | Table   | Statement                                      | Timing | Created | sql_mode                 | Definer        | character_set_client | collation_connection | Database Collation |
+----------------+--------+---------+------------------------------------------------+--------+---------+--------------------------+----------------+----------------------+----------------------+--------------------+
| archive_insert | INSERT | archive | SET NEW.ar_comment = '', NEW.ar_comment_id = 0 | BEFORE | NULL    | IGNORE_BAD_TABLE_OPTIONS | root@localhost | utf8                 | utf8_general_ci      | binary             |
| archive_update | UPDATE | archive | SET NEW.ar_comment = '', NEW.ar_comment_id = 0 | BEFORE | NULL    | IGNORE_BAD_TABLE_OPTIONS | root@localhost | utf8                 | utf8_general_ci      | binary             |
+----------------+--------+---------+------------------------------------------------+--------+---------+--------------------------+----------------+----------------------+----------------------+--------------------+

The rest of the triggers should be checked just in case.

  • s1
    • codfw
    •   eqiad
  • s2
    • codfw
    •   eqiad
  • s3
    • codfw
    •   eqiad
  • s4
    • codfw
    •   eqiad
  • s5
    • codfw
    •   eqiad
  • s6
    • codfw
    •   eqiad
  • s7
    • codfw
    •   eqiad
  • s8
    • codfw
    •   eqiad

Event Timeline

Marostegui triaged this task as Medium priority.Oct 5 2019, 4:59 AM
Marostegui moved this task from Triage to In progress on the DBA board.

Mentioned in SAL (#wikimedia-operations) [2019-10-07T05:08:38Z] <marostegui> Stop replication on db2076 to modify triggers on db2096:3316 T234704

Mentioned in SAL (#wikimedia-operations) [2019-10-07T05:10:39Z] <marostegui> The above was for db2095:3316 T234704

Mentioned in SAL (#wikimedia-operations) [2019-10-10T11:35:47Z] <marostegui> Stop replication on db2077 to change triggers on db2095:3317 - T234704

Mentioned in SAL (#wikimedia-operations) [2019-10-16T05:11:32Z] <marostegui> Change s2 triggers for archive table from db1125:3312 T234704

Mentioned in SAL (#wikimedia-operations) [2019-10-16T05:14:55Z] <marostegui> Change s7 triggers for archive table from db1125:3317 T234704

This comment was removed by Marostegui.

Mentioned in SAL (#wikimedia-operations) [2019-11-12T06:44:00Z] <marostegui> Change triggers on s5 db2094 - T234704

Mentioned in SAL (#wikimedia-operations) [2019-11-14T08:42:22Z] <marostegui> Remove ar_comment from triggers on db1124:3315 - T234704

Mentioned in SAL (#wikimedia-operations) [2019-11-18T09:11:49Z] <marostegui> Remove ar_comment from triggers on db2094:3318 - T234704

Mentioned in SAL (#wikimedia-operations) [2019-11-25T14:38:10Z] <marostegui> Remove triggers from archive table on s1 codfw sanitarium T234704

Mentioned in SAL (#wikimedia-operations) [2019-11-26T06:44:05Z] <marostegui> Remove triggers for ar_comment on db1124:3318 T234704

Mentioned in SAL (#wikimedia-operations) [2019-11-28T13:31:38Z] <marostegui> Remove ar_comment triggers from db1124:3311 for enwiki.archive - T234704

Mentioned in SAL (#wikimedia-operations) [2019-11-29T09:33:53Z] <marostegui> Remove triggers from db2094:3313 - T234704

Mentioned in SAL (#wikimedia-operations) [2019-12-03T05:47:22Z] <marostegui> Remove ar_comment triggers from s3 db1124:3313 - T234704

Mentioned in SAL (#wikimedia-operations) [2019-12-10T06:06:44Z] <marostegui> Remove triggers from db2095:3314 for ar_comment - T234704

Mentioned in SAL (#wikimedia-operations) [2019-12-16T06:29:13Z] <marostegui> Remove triggers for ar_comment on db1125:3314 T234704

Change 557808 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] filtered_tables: Remove ar_comment reference

https://gerrit.wikimedia.org/r/557808

Change 557808 merged by Marostegui:
[operations/puppet@production] filtered_tables: Remove ar_comment reference

https://gerrit.wikimedia.org/r/557808

Marostegui updated the task description. (Show Details)

All done