Page MenuHomePhabricator

Sort out replication error on frdb2002
Closed, ResolvedPublic

Description

Replication hung on frdb2002 tonight with the following error:

Last_SQL_Error: Could not execute Update_rows_v1 event on table civicrm.civicrm_contribution_recur; Can't find record in 'civicrm_contribution_recur', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log frdb1004-bin.000228, end_log_pos 26835108

Sort out what happened here and figure out if the transactions can be recovered or if we need to reclone from frdb2001.

Event Timeline

Dwisehaupt moved this task from Triage to In Progress on the fundraising-tech-ops board.

After digging in, I found the issue. There was one transaction that was skipped during the utf8mb4 testing and other work on frdb2002. When trying to reapply that transaction, I hit a snag with dupe keys on the log tables. This prompted the thought that the log tables may be 'off by one' on this host and could thus cause issues in the future. The cleanest and safest route is to reclone from frdb2001. This recloning has started.

Dwisehaupt moved this task from In Progress to Done on the fundraising-tech-ops board.

DB is recloned. It and the source db (frdb2001) are back in action.