Page MenuHomePhabricator

66 rows from external storage (dewiki) gave duplicate key errors on master failover
Closed, ResolvedPublic

Description

On external storage es2 failover, 66 rows written to dewiki es1015 gave a duplicate key error. The rows were deleted only on es1015 (not on the rest of servers of the shard), and then replication continued.

The rows were backed up on es2002. We checked the text tables on dewiki and we saw no duplicate references to those rows or orphan revisions however, so probably those new revisions were discarded and inserted with other ids, but a deeper analysis probably is required to make sure no data was loss.

Event Timeline

jcrespo created this task.Dec 5 2016, 11:39 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptDec 5 2016, 11:39 AM
1978Gage2001 moved this task from Triage to In progress on the DBA board.Dec 11 2017, 9:45 AM
Marostegui moved this task from In progress to Triage on the DBA board.Dec 11 2017, 11:07 AM
Marostegui closed this task as Resolved.Mar 13 2018, 10:45 AM
Marostegui claimed this task.
Marostegui added a subscriber: Marostegui.

As I am with the "checksumming" state of mind I have also taken care of this.
The table is consistent across all the eqiad hosts.