db1124 reported broken replication 2018-11-14 09:42
wikidatawiki.pagelinks (s8) was not able to execute an event because of a missing row.
We searched the event which was not able to execute, it was a DELETE ROW, but the row was non-existent on the santitarium host.
We created the row manually and restarted replication
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | • Banyek | T209521 replication broken on db1124:3318 on wikidata.pagelinks | |||
| Resolved | Marostegui | T212574 Recheck wikidatawiki.pagelinks table across all hosts in s8 |
This broke again yesterday:
Dec 06 18:10:48 db1124 mysqld[3110]: 2018-12-06 18:10:48 140545623860992 [ERROR] Slave SQL: Could not execute Delete_rows_v1 event on table wikidatawiki.pagelinks; Can't find record in 'pagelinks', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master l Dec 06 18:10:48 db1124 mysqld[3110]: 2018-12-06 18:10:48 140545623860992 [Warning] Slave: Can't find record in 'pagelinks' Error_code: 1032 Dec 06 18:10:48 db1124 mysqld[3110]: 2018-12-06 18:10:48 140545623860992 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'db1087-bin.003545' position 943920625 Dec 06 18:10:48 db1124 mysqld[3110]: 2018-12-06 18:10:48 140545623860992 [Note] Slave SQL thread exiting, replication stopped in log 'db1087-bin.003545' at position 943920625
I fixed it with:
INSERT INTO `pagelinks` VALUES (7232144,0,'Q639669',0);
This has happaned today again with pagelinks. I have had to add a few rows and delete multiple rows after it that were giving duplicate entry for another pl_from value.
I have waited until db1124:3318 fully caught up with the master before I left.
We need to check again pagelinks between the master and the sanitarium host.
Another breakage happened today:
05:28 <+icinga-wm> PROBLEM - MariaDB Slave SQL: s8 on db1124 is CRITICAL: CRITICAL slave_sql_state Slave_SQL_Running: No, Errno: 1032, Errmsg: Could not execute Delete_rows_v1 event on table wikidatawiki.pagelinks: Cant find record in pagelinks, Error_code: 1032: handler error HA_ERR_KEY_NOT_FOUND: the events master log db1087-bin.003648, end_log_pos 920552713 05:38 <+icinga-wm> PROBLEM - MariaDB Slave Lag: s8 on db1124 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 750.52 seconds