Create a cookbook to copy data between WDQS servers
Closed, ResolvedPublic
Actions

Description

In some cases, we want to copy data between wdqs servers. Either after a re-image, or in case of data corruption (which we had recently).

The procedure is documented, but not yet fully automated. A Spicerack Cookbook seems like the way to go for automation.

Details

Subject	Repo	Branch	Lines +/-
wdqs: add some logging to data transfer cookbook	operations/cookbooks	master	+10 -0
wdqs: fix source and destination that were inverted	operations/cookbooks	master	+2 -2
wdqs: ensure receiver has started before sending file	operations/cookbooks	master	+2 -0
wdqs: add arguments to not use LVS	operations/cookbooks	master	+6 -3
Add wdqs data transfer cookbook	operations/cookbooks	master	+114 -0

Customize query in gerrit

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Open		None	T203943 Spicerack cookbooks TODO list
		Resolved		• Mathew.onipe	T213401 Create a cookbook to copy data between WDQS servers

Event Timeline

Gehel created this task.Jan 10 2019, 9:43 AM

Restricted Application added a project: Wikidata. · View Herald TranscriptJan 10 2019, 9:43 AM

Restricted Application added a subscriber: Aklapper. · View Herald Transcript

herron triaged this task as Medium priority.Jan 10 2019, 5:56 PM

herron added a project: SRE-tools.

herron added a subscriber: Volans.

Addshore moved this task from incoming to monitoring on the Wikidata board.Jan 11 2019, 10:05 AM

Smalyshev moved this task from Incoming to Operations/SRE on the Wikidata-Query-Service board.Jan 12 2019, 6:31 AM

• Mathew.onipe claimed this task.Jan 16 2019, 10:08 AM

• Mathew.onipe added a project: Discovery-Search (Current work).

Volans added a parent task: T203943: Spicerack cookbooks TODO list.Jan 24 2019, 9:44 PM

• Phabricator_maintenance moved this task from Backlog to Acknowledged on the SRE board.Jan 26 2019, 11:02 PM

TJones edited projects, added Discovery-Search; removed Discovery-Search (Current work).Jan 29 2019, 9:57 PM

TJones moved this task from needs triage to Ops / SRE on the Discovery-Search board.

• Mathew.onipe edited projects, added Discovery-Search (Current work); removed Discovery-Search.Feb 6 2019, 8:34 AM

Change 488256 had a related patch set uploaded (by Mathew.onipe; owner: Mathew.onipe):
[operations/cookbooks@master] Add wdqs data transfer cookbook

https://gerrit.wikimedia.org/r/488256

gerritbot added a project: Patch-For-Review.Feb 6 2019, 8:36 AM

• Mathew.onipe moved this task from Incoming to Needs review on the Discovery-Search (Current work) board.Feb 19 2019, 6:33 PM

• Mathew.onipe edited projects, added Discovery-Search; removed Discovery-Search (Current work).Apr 2 2019, 5:21 PM

Change 488256 had a related patch set uploaded (by Mathew.onipe; owner: Mathew.onipe):
[operations/cookbooks@master] Add wdqs data transfer cookbook

https://gerrit.wikimedia.org/r/488256

• Mathew.onipe edited projects, added Discovery-Search (Current work); removed Discovery-Search.Apr 14 2019, 8:33 AM

Change 488256 merged by Gehel:
[operations/cookbooks@master] Add wdqs data transfer cookbook

https://gerrit.wikimedia.org/r/488256

Change 504327 had a related patch set uploaded (by Gehel; owner: Gehel):
[operations/cookbooks@master] wdqs: add arguments to not use LVS

https://gerrit.wikimedia.org/r/504327

Change 504327 merged by Gehel:
[operations/cookbooks@master] wdqs: add arguments to not use LVS

https://gerrit.wikimedia.org/r/504327

Change 504332 had a related patch set uploaded (by Gehel; owner: Gehel):
[operations/cookbooks@master] wdqs: ensure reaceiver has started before sending file

https://gerrit.wikimedia.org/r/504332

Change 504332 merged by Gehel:
[operations/cookbooks@master] wdqs: ensure receiver has started before sending file

https://gerrit.wikimedia.org/r/504332

Change 504333 had a related patch set uploaded (by Gehel; owner: Gehel):
[operations/cookbooks@master] wdqs: fix source and destination that were inverted

https://gerrit.wikimedia.org/r/504333

Change 504333 merged by Gehel:
[operations/cookbooks@master] wdqs: fix source and destination that were inverted

https://gerrit.wikimedia.org/r/504333

Change 504343 had a related patch set uploaded (by Gehel; owner: Gehel):
[operations/cookbooks@master] wdqs: add some logging to data transfer cookbook

https://gerrit.wikimedia.org/r/504343

Change 504343 merged by Gehel:
[operations/cookbooks@master] wdqs: add some logging to data transfer cookbook

https://gerrit.wikimedia.org/r/504343

• Mathew.onipe moved this task from Needs review to Needs Reporting on the Discovery-Search (Current work) board.Apr 16 2019, 5:22 PM

• Mathew.onipe moved this task from Needs Reporting to Incoming on the Discovery-Search (Current work) board.

• Mathew.onipe moved this task from Incoming to Needs Reporting on the Discovery-Search (Current work) board.Apr 18 2019, 8:32 AM

debt closed this task as Resolved.Apr 19 2019, 8:52 PM

joanna_borun added a project: Spicerack.Jun 15 2022, 10:43 AM

Restricted Application added a project: Infrastructure-Foundations. · View Herald TranscriptJun 15 2022, 10:43 AM

• Mathew.onipe mentioned this in rCCKB4b9e9795aa64: Add wdqs data transfer cookbook.Dec 14 2022, 3:24 PM

Gehel mentioned this in rCCKB4de3b540e2d4: wdqs: add arguments to not use LVS.Dec 14 2022, 3:24 PM

Gehel mentioned this in rCCKBb311175ffe15: wdqs: ensure receiver has started before sending file.