Page MenuHomePhabricator

Reload WDQS dataset
Closed, ResolvedPublic

Description

After T121274 and T175948 are merged and deployed, we will need to reload the data for all servers. If we want to do any other data modifications, we may attach them to this ticket too. Dependencies:

Event Timeline

Restricted Application added projects: Wikidata, Discovery. · View Herald TranscriptSep 25 2017, 2:25 AM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript

For T175948, we probably will need a more complex reload procedure, since we need config change and old data is incompatible with new config. So we may want to do this:

  1. Commit new code & config but do not deploy it
  2. On each server: shut it down, deploy new code & config, load new data, start it again

We could maybe do two codfw servers at the same time due to lower traffic they are getting.

Smalyshev triaged this task as Normal priority.Oct 10 2017, 5:25 PM

According to T178180, the external IDs will be enabled on Oct 26th. That means the dump of Oct 30-31th should contain the necessary data, and we can start the reloads on the week starting Oct 30th.

Smalyshev updated the task description. (Show Details)Oct 31 2017, 11:53 PM
Smalyshev moved this task from Backlog to Doing on the User-Smalyshev board.Nov 7 2017, 8:01 AM

Mentioned in SAL (#wikimedia-operations) [2017-11-10T18:08:04Z] <smalyshev@tin> Started deploy [wdqs/wdqs@ccab8ce]: data reload/T176593

Mentioned in SAL (#wikimedia-operations) [2017-11-10T18:08:39Z] <smalyshev@tin> Finished deploy [wdqs/wdqs@ccab8ce]: data reload/T176593 (duration: 00m 34s)

Change 390440 had a related patch set uploaded (by Gehel; owner: Gehel):
[operations/puppet@production] wdqs: allow port 9876 inside the wdqs clsuter for netcat file transfer

https://gerrit.wikimedia.org/r/390440

Change 390440 merged by Gehel:
[operations/puppet@production] wdqs: allow port 9876 inside the wdqs clsuter for netcat file transfer

https://gerrit.wikimedia.org/r/390440

Mentioned in SAL (#wikimedia-operations) [2017-11-13T18:28:33Z] <gehel> deploy latest blazegraph + GUI on wdqs200[23] to switch vocabulary - T176593

Mentioned in SAL (#wikimedia-operations) [2017-11-13T19:20:27Z] <smalyshev@tin> Started deploy [wdqs/wdqs@b44cf27]: data reload/T176593

Mentioned in SAL (#wikimedia-operations) [2017-11-13T19:20:47Z] <smalyshev@tin> Finished deploy [wdqs/wdqs@b44cf27]: data reload/T176593 (duration: 00m 20s)

Mentioned in SAL (#wikimedia-operations) [2017-11-14T17:42:03Z] <smalyshev@tin> Started deploy [wdqs/wdqs@b44cf27]: data reload/T176593

Mentioned in SAL (#wikimedia-operations) [2017-11-14T17:42:19Z] <smalyshev@tin> Finished deploy [wdqs/wdqs@b44cf27]: data reload/T176593 (duration: 00m 16s)

Change 391630 had a related patch set uploaded (by Gehel; owner: Gehel):
[operations/puppet@production] wdqs: move ldf traffic to wdqs1004

https://gerrit.wikimedia.org/r/391630

Change 391630 merged by Gehel:
[operations/puppet@production] wdqs: move ldf traffic to wdqs1004

https://gerrit.wikimedia.org/r/391630

Smalyshev closed this task as Resolved.Nov 16 2017, 12:10 AM

Reload is now done.

Mentioned in SAL (#wikimedia-operations) [2017-11-16T06:14:37Z] <smalyshev@tin> Started deploy [wdqs/wdqs@b44cf27]: data reload/T176593

Mentioned in SAL (#wikimedia-operations) [2017-11-16T06:15:04Z] <smalyshev@tin> Finished deploy [wdqs/wdqs@b44cf27]: data reload/T176593 (duration: 00m 28s)