Page MenuHomePhabricator
Create Task
Maniphest T234704

Remove ar_comment from sanitarium triggers
Closed, ResolvedPublic
Actions

Description

While the schema change arrives to the sanitarium hosts, replication breaks due to missing column ar_comment on the sanitization triggers:

Last_SQL_Error: Could not execute Write_rows_v1 event on table jawiki.archive; Unknown column 'ar_comment' in 'NEW', Error_code: 1054; handler error HA_ERR_GENERIC; the event's master log db2076-bin.001830, end_log_pos 827488672
root@cumin1001:/home/marostegui/T233135# mysql.py -hdb2095:3316 -e "show triggers like 'archive%'" jawiki
+----------------+--------+---------+------------------------------------------------+--------+---------+--------------------------+----------------+----------------------+----------------------+--------------------+
| Trigger        | Event  | Table   | Statement                                      | Timing | Created | sql_mode                 | Definer        | character_set_client | collation_connection | Database Collation |
+----------------+--------+---------+------------------------------------------------+--------+---------+--------------------------+----------------+----------------------+----------------------+--------------------+
| archive_insert | INSERT | archive | SET NEW.ar_comment = '', NEW.ar_comment_id = 0 | BEFORE | NULL    | IGNORE_BAD_TABLE_OPTIONS | root@localhost | utf8                 | utf8_general_ci      | binary             |
| archive_update | UPDATE | archive | SET NEW.ar_comment = '', NEW.ar_comment_id = 0 | BEFORE | NULL    | IGNORE_BAD_TABLE_OPTIONS | root@localhost | utf8                 | utf8_general_ci      | binary             |
+----------------+--------+---------+------------------------------------------------+--------+---------+--------------------------+----------------+----------------------+----------------------+--------------------+

The rest of the triggers should be checked just in case.

  • s1
    • codfw
    •   eqiad
  • s2
    • codfw
    •   eqiad
  • s3
    • codfw
    •   eqiad
  • s4
    • codfw
    •   eqiad
  • s5
    • codfw
    •   eqiad
  • s6
    • codfw
    •   eqiad
  • s7
    • codfw
    •   eqiad
  • s8
    • codfw
    •   eqiad

Details

ProjectBranchLines +/-Subject
operations/puppetproduction+0 -1filtered_tables: Remove ar_comment reference
Customize query in gerrit

Related Objects
Search...

Event Timeline

Marostegui created this task.Oct 5 2019, 4:59 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 5 2019, 4:59 AM
Marostegui triaged this task as Medium priority.Oct 5 2019, 4:59 AM
Marostegui moved this task from Triage to In progress on the DBA board.
Stashbot added a comment.Oct 7 2019, 5:08 AM

Mentioned in SAL (#wikimedia-operations) [2019-10-07T05:08:38Z] <marostegui> Stop replication on db2076 to modify triggers on db2096:3316 T234704

Stashbot added a comment.Oct 7 2019, 5:10 AM

Mentioned in SAL (#wikimedia-operations) [2019-10-07T05:10:39Z] <marostegui> The above was for db2095:3316 T234704

Marostegui updated the task description. (Show Details)Oct 7 2019, 5:20 AM
Marostegui updated the task description. (Show Details)Oct 7 2019, 5:44 AM
Stashbot added a comment.Oct 10 2019, 11:35 AM

Mentioned in SAL (#wikimedia-operations) [2019-10-10T11:35:47Z] <marostegui> Stop replication on db2077 to change triggers on db2095:3317 - T234704

Marostegui updated the task description. (Show Details)Oct 10 2019, 11:41 AM
Stashbot added a comment.Oct 16 2019, 5:11 AM

Mentioned in SAL (#wikimedia-operations) [2019-10-16T05:11:32Z] <marostegui> Change s2 triggers for archive table from db1125:3312 T234704

Stashbot added a comment.Oct 16 2019, 5:14 AM

Mentioned in SAL (#wikimedia-operations) [2019-10-16T05:14:55Z] <marostegui> Change s7 triggers for archive table from db1125:3317 T234704

Marostegui updated the task description. (Show Details)Oct 16 2019, 5:15 AM
This comment was removed by Marostegui.
Stashbot mentioned this in T234066: Schema change to rename user_newtalk indexes.Oct 16 2019, 10:17 AM
Stashbot mentioned this in T233135: Schema change for refactored actor and comment storage.

Mentioned in SAL (#wikimedia-operations) [2019-10-16T10:17:10Z] <marostegui> Stop replication on s2 codfw master for schema change and to modify sanitarium triggers T234066 T233135 T234704

Marostegui updated the task description. (Show Details)Oct 16 2019, 10:21 AM
Stashbot added a comment.Nov 12 2019, 6:44 AM

Mentioned in SAL (#wikimedia-operations) [2019-11-12T06:44:00Z] <marostegui> Change triggers on s5 db2094 - T234704

Marostegui updated the task description. (Show Details)Nov 12 2019, 6:44 AM
Stashbot added a comment.Nov 14 2019, 8:42 AM

Mentioned in SAL (#wikimedia-operations) [2019-11-14T08:42:22Z] <marostegui> Remove ar_comment from triggers on db1124:3315 - T234704

Marostegui updated the task description. (Show Details)Nov 14 2019, 8:51 AM
Stashbot added a comment.Nov 18 2019, 9:11 AM

Mentioned in SAL (#wikimedia-operations) [2019-11-18T09:11:49Z] <marostegui> Remove ar_comment from triggers on db2094:3318 - T234704

Marostegui updated the task description. (Show Details)Nov 18 2019, 9:15 AM
Stashbot added a comment.Nov 25 2019, 2:38 PM

Mentioned in SAL (#wikimedia-operations) [2019-11-25T14:38:10Z] <marostegui> Remove triggers from archive table on s1 codfw sanitarium T234704

Marostegui updated the task description. (Show Details)Nov 25 2019, 2:39 PM
Stashbot added a comment.Nov 26 2019, 6:44 AM

Mentioned in SAL (#wikimedia-operations) [2019-11-26T06:44:05Z] <marostegui> Remove triggers for ar_comment on db1124:3318 T234704

Marostegui updated the task description. (Show Details)Nov 26 2019, 6:46 AM
Stashbot added a comment.Nov 28 2019, 1:31 PM

Mentioned in SAL (#wikimedia-operations) [2019-11-28T13:31:38Z] <marostegui> Remove ar_comment triggers from db1124:3311 for enwiki.archive - T234704

Marostegui updated the task description. (Show Details)Nov 28 2019, 1:32 PM
Stashbot added a comment.Nov 29 2019, 9:33 AM

Mentioned in SAL (#wikimedia-operations) [2019-11-29T09:33:53Z] <marostegui> Remove triggers from db2094:3313 - T234704

Marostegui updated the task description. (Show Details)Nov 29 2019, 9:44 AM
Marostegui updated the task description. (Show Details)Dec 2 2019, 3:00 PM
Stashbot added a comment.Dec 3 2019, 5:47 AM

Mentioned in SAL (#wikimedia-operations) [2019-12-03T05:47:22Z] <marostegui> Remove ar_comment triggers from s3 db1124:3313 - T234704

Marostegui updated the task description. (Show Details)Dec 3 2019, 5:48 AM
Stashbot added a comment.Dec 10 2019, 6:06 AM

Mentioned in SAL (#wikimedia-operations) [2019-12-10T06:06:44Z] <marostegui> Remove triggers from db2095:3314 for ar_comment - T234704

Marostegui updated the task description. (Show Details)Dec 10 2019, 6:08 AM
Stashbot added a comment.Dec 16 2019, 6:29 AM

Mentioned in SAL (#wikimedia-operations) [2019-12-16T06:29:13Z] <marostegui> Remove triggers for ar_comment on db1125:3314 T234704

Marostegui updated the task description. (Show Details)Dec 16 2019, 7:41 AM
gerritbot added a comment.Dec 16 2019, 7:43 AM

Change 557808 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] filtered_tables: Remove ar_comment reference

https://gerrit.wikimedia.org/r/557808

gerritbot added a project: Patch-For-Review.Dec 16 2019, 7:43 AM
gerritbot added a comment.Dec 16 2019, 8:52 AM

Change 557808 merged by Marostegui:
[operations/puppet@production] filtered_tables: Remove ar_comment reference

https://gerrit.wikimedia.org/r/557808

Marostegui closed this task as Resolved.Dec 16 2019, 8:53 AM
Marostegui updated the task description. (Show Details)

All done

Maintenance_bot removed a project: Patch-For-Review.Dec 16 2019, 9:10 AM
Content licensed under Creative Commons Attribution-ShareAlike 3.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL