Page MenuHomePhabricator
Create Task
Maniphest T238399

Reimport wikidatawiki.{pagelinks,page} on labsdb1010
Closed, ResolvedPublic
Actions

Description

Due to a recent crash on labsdb1010 (T233986#5527906) replication broke this week on wikidatawiki.pagelinks: T233986#5659184
I fixed that issue manually and ran a check against labsdb1011 and it has revealed more differences on pagelinks table.

This table should be reimported from labsdb1011 (or labsdb1012 as that one is only in use during the first days of the month).

I am also checking other main tables just in case there are more differences due to that crash.

Update: page table also has differences

Re-import progress:

  • page
  • pagelinks

Details

ProjectBranchLines +/-Subject
operations/puppetproduction+4 -4dbproxy: Depool labsdb1010
operations/puppetproduction+2 -2dbproxy1011: Depool labsdb1010
Customize query in gerrit

Event Timeline

Marostegui created this task.Nov 15 2019, 9:08 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptNov 15 2019, 9:08 AM
Marostegui claimed this task.Nov 15 2019, 9:10 AM
Marostegui triaged this task as Medium priority.
Marostegui moved this task from Triage to Pending comment on the DBA board.

I will try to do this next week using mydumper (the table is 91GB) - I will have labsdb1010 depooled during the re-import time.
We will probably have to drop+create the views I guess. We'll see.

Marostegui renamed this task from Reimport wikidatawiki.pagelinks on labsdb1010 to Reimport wikidatawiki.{pagelinks,page} on labsdb1010.Nov 15 2019, 1:22 PM
Marostegui updated the task description. (Show Details)
Marostegui updated the task description. (Show Details)Nov 22 2019, 8:45 AM
Stashbot added a comment.Nov 27 2019, 9:01 AM

Mentioned in SAL (#wikimedia-operations) [2019-11-27T09:01:16Z] <marostegui> Stop replication on 1124:3318 to reimport wikidatawiki.page table on labsdb1010 - T238399

gerritbot added a comment.Dec 4 2019, 7:07 AM

Change 554420 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] dbproxy1011: Depool labsdb1010

https://gerrit.wikimedia.org/r/554420

gerritbot added a project: Patch-For-Review.Dec 4 2019, 7:07 AM

Change 554420 merged by Marostegui:
[operations/puppet@production] dbproxy1011: Depool labsdb1010

https://gerrit.wikimedia.org/r/554420

Stashbot added a comment.Dec 4 2019, 7:09 AM

Mentioned in SAL (#wikimedia-operations) [2019-12-04T07:09:30Z] <marostegui> Depool labsdb1010 to reimport wikidatawiki.page - T238399

Maintenance_bot removed a project: Patch-For-Review.Dec 4 2019, 7:10 AM
Marostegui added a comment.Dec 4 2019, 8:58 AM

wikidatawiki.page has been reimported from labsdb1012.
Views for that table have been recreated as well.

Marostegui updated the task description. (Show Details)Dec 4 2019, 8:58 AM
Marostegui moved this task from Pending comment to In progress on the DBA board.
Stashbot added a comment.Dec 4 2019, 9:15 AM

Mentioned in SAL (#wikimedia-operations) [2019-12-04T09:15:33Z] <marostegui> Reload labsdb1010 after reimporting wikidatawiki.page - T238399

gerritbot added a comment.Dec 16 2019, 6:25 AM

Change 557788 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] dbproxy: Depool labsdb1010

https://gerrit.wikimedia.org/r/557788

gerritbot added a project: Patch-For-Review.Dec 16 2019, 6:25 AM
gerritbot added a comment.Dec 16 2019, 6:41 AM

Change 557788 merged by Marostegui:
[operations/puppet@production] dbproxy: Depool labsdb1010

https://gerrit.wikimedia.org/r/557788

Stashbot added a comment.Dec 16 2019, 6:42 AM

Mentioned in SAL (#wikimedia-operations) [2019-12-16T06:42:16Z] <marostegui> Depool labsdb1010 - T238399

Maintenance_bot removed a project: Patch-For-Review.Dec 16 2019, 7:10 AM
Stashbot added a comment.Dec 16 2019, 7:11 AM

Mentioned in SAL (#wikimedia-operations) [2019-12-16T07:11:37Z] <marostegui> Stop replication in the same position in labsdb1010 and labsdb1012 - T238399

Marostegui added a comment.Dec 16 2019, 9:13 AM
In T238399#5743379, @Stashbot wrote:

Mentioned in SAL (#wikimedia-operations) [2019-12-16T07:11:37Z] <marostegui> Stop replication in the same position in labsdb1010 and labsdb1012 - T238399

I have postponed this until an alter table is finished, so mydumper can get the global lock needed - so the message above can be dismissed, replication is restarted and I will log again once I attempt this reimport

Stashbot added a comment.Dec 16 2019, 9:50 AM

Mentioned in SAL (#wikimedia-operations) [2019-12-16T09:50:47Z] <marostegui> Stop replication in the same position in labsdb1010 and labsdb1012 - T238399

Marostegui added a comment.Dec 17 2019, 6:39 AM

wikidatawiki.pagelinks has been imported into labsdb1010 from labsdb1012 - the host is now catching up.

Marostegui updated the task description. (Show Details)Dec 17 2019, 6:39 AM
Stashbot added a comment.Dec 17 2019, 8:31 AM

Mentioned in SAL (#wikimedia-operations) [2019-12-17T08:31:04Z] <marostegui> Repool labsdb1010 T238399

Marostegui closed this task as Resolved.Dec 17 2019, 8:31 AM

Host repooled

Content licensed under Creative Commons Attribution-ShareAlike 3.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL