Page MenuHomePhabricator

DB reload for WDQS
Closed, ResolvedPublic

Description

This task is for reloading WDQS database from scratch. The plan for hosts is as follows:

  • 1010
  • 1009 - currently loading alternative version (with rawRecords enabled)
  • 2004
  • 2001
  • 1003
  • 1004
  • 2005
  • 2006
  • 1007
  • 1008
  • 2002
  • 2003
  • 1005
  • 1006

Only one machine per cluster (public or internal) should be loaded at the same time, but I think it's ok to load one public and one internal one in parallel.

Related Objects

StatusAssignedTask
OpenNone
ResolvedGehel

Event Timeline

Restricted Application added a project: Wikidata. · View Herald TranscriptJul 16 2019, 12:44 AM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
Smalyshev triaged this task as High priority.Jul 16 2019, 12:44 AM

Change 523266 had a related patch set uploaded (by Smalyshev; owner: Gehel):
[operations/puppet@production] wdqs: introduced tuned journal options to wdqs2001.

https://gerrit.wikimedia.org/r/523266

Smalyshev added a comment.EditedJul 16 2019, 5:54 AM

Hmm I am seeing some missing items in wdqs1010, so let's not update for now until I find out what's up.

@Gehel Never mind, this was me misconfiguring the query to use wrong namespace :\ The data is fine, so we can proceed.

Mentioned in SAL (#wikimedia-operations) [2019-07-16T18:54:56Z] <gehel> data copy from wdqs2004 to wdqs2001 - T228122

Change 523266 merged by Gehel:
[operations/puppet@production] wdqs: introduced tuned journal options to wdqs2001.

https://gerrit.wikimedia.org/r/523266

Smalyshev updated the task description. (Show Details)Jul 16 2019, 8:56 PM
Smalyshev updated the task description. (Show Details)

2001 seems to be doing fine, so we can do the next set I think.

Change 523866 had a related patch set uploaded (by Gehel; owner: Gehel):
[operations/puppet@production] wdqs: introduced tuned journal options to wdqs1003.

https://gerrit.wikimedia.org/r/523866

Change 523867 had a related patch set uploaded (by Gehel; owner: Gehel):
[operations/puppet@production] wdqs: introduced tuned journal options to wdqs1004.

https://gerrit.wikimedia.org/r/523867

Change 523868 had a related patch set uploaded (by Gehel; owner: Gehel):
[operations/puppet@production] wdqs: introduced tuned journal options to wdqs2005.

https://gerrit.wikimedia.org/r/523868

Change 523869 had a related patch set uploaded (by Gehel; owner: Gehel):
[operations/puppet@production] wdqs: introduced tuned journal options to wdqs2006.

https://gerrit.wikimedia.org/r/523869

Change 523870 had a related patch set uploaded (by Gehel; owner: Gehel):
[operations/puppet@production] wdqs: introduced tuned journal options to wdqs1007.

https://gerrit.wikimedia.org/r/523870

Change 523871 had a related patch set uploaded (by Gehel; owner: Gehel):
[operations/puppet@production] wdqs: introduced tuned journal options to wdqs1008.

https://gerrit.wikimedia.org/r/523871

Change 523872 had a related patch set uploaded (by Gehel; owner: Gehel):
[operations/puppet@production] wdqs: introduced tuned journal options to wdqs2002.

https://gerrit.wikimedia.org/r/523872

Change 523873 had a related patch set uploaded (by Gehel; owner: Gehel):
[operations/puppet@production] wdqs: introduced tuned journal options to wdqs2003.

https://gerrit.wikimedia.org/r/523873

Change 523874 had a related patch set uploaded (by Gehel; owner: Gehel):
[operations/puppet@production] wdqs: introduced tuned journal options to wdqs1005.

https://gerrit.wikimedia.org/r/523874

Change 523875 had a related patch set uploaded (by Gehel; owner: Gehel):
[operations/puppet@production] wdqs: introduced tuned journal options to wdqs1006.

https://gerrit.wikimedia.org/r/523875

Change 523866 merged by Gehel:
[operations/puppet@production] wdqs: introduced tuned journal options to wdqs1003.

https://gerrit.wikimedia.org/r/523866

Gehel updated the task description. (Show Details)Jul 17 2019, 11:55 AM

Change 523867 merged by Gehel:
[operations/puppet@production] wdqs: introduced tuned journal options to wdqs1004.

https://gerrit.wikimedia.org/r/523867

Gehel updated the task description. (Show Details)Jul 17 2019, 1:34 PM

Change 523868 merged by Gehel:
[operations/puppet@production] wdqs: introduced tuned journal options to wdqs2005.

https://gerrit.wikimedia.org/r/523868

Gehel updated the task description. (Show Details)Jul 17 2019, 3:20 PM

Change 523869 merged by Gehel:
[operations/puppet@production] wdqs: introduced tuned journal options to wdqs2006.

https://gerrit.wikimedia.org/r/523869

Gehel updated the task description. (Show Details)Jul 18 2019, 9:27 AM

Change 523870 merged by Gehel:
[operations/puppet@production] wdqs: introduced tuned journal options to wdqs1007.

https://gerrit.wikimedia.org/r/523870

Change 523871 merged by Gehel:
[operations/puppet@production] wdqs: introduced tuned journal options to wdqs1008.

https://gerrit.wikimedia.org/r/523871

Gehel updated the task description. (Show Details)Jul 18 2019, 12:49 PM

Change 523872 merged by Gehel:
[operations/puppet@production] wdqs: introduced tuned journal options to wdqs2002.

https://gerrit.wikimedia.org/r/523872

Gehel updated the task description. (Show Details)Jul 18 2019, 2:09 PM

Change 523873 merged by Gehel:
[operations/puppet@production] wdqs: introduced tuned journal options to wdqs2003.

https://gerrit.wikimedia.org/r/523873

Gehel updated the task description. (Show Details)Jul 18 2019, 3:22 PM
Dzahn added a subscriber: Dzahn.Jul 18 2019, 9:35 PM

Icinga says for wdqs2004: "PYBAL CRITICAL - CRITICAL - wdqs-internal_80: Servers wdqs2004.codfw.wmnet are marked down but pooled"

and for wdqs1010: "PROCS CRITICAL: 0 processes with UID = 499 (blazegraph), regex args '^java .* --port 9999 .* blazegraph-service-.*war'"

Mentioned in SAL (#wikimedia-operations) [2019-07-19T08:22:57Z] <gehel> repooling wdqs2003 - T228122

Mentioned in SAL (#wikimedia-operations) [2019-07-19T08:24:02Z] <gehel> repooling wdqs2004 - T228122

Gehel updated the task description. (Show Details)Jul 19 2019, 12:29 PM

Change 523874 merged by Gehel:
[operations/puppet@production] wdqs: introduced tuned journal options to wdqs1005.

https://gerrit.wikimedia.org/r/523874

Gehel updated the task description. (Show Details)Jul 19 2019, 4:20 PM

Change 523875 merged by Gehel:
[operations/puppet@production] wdqs: introduced tuned journal options to wdqs1006.

https://gerrit.wikimedia.org/r/523875

Smalyshev closed this task as Resolved.Jul 21 2019, 8:01 PM
Smalyshev updated the task description. (Show Details)

Change 524954 had a related patch set uploaded (by Smalyshev; owner: Smalyshev):
[operations/puppet@production] Update WDQS standard settings to new DB settings

https://gerrit.wikimedia.org/r/524954

Mentioned in SAL (#wikimedia-operations) [2019-07-25T17:18:15Z] <volans> disabled puppet on A:wdqs-all, deploying gerrit/524954 - T228122

Change 524954 merged by Volans:
[operations/puppet@production] Update WDQS standard settings to new DB settings

https://gerrit.wikimedia.org/r/524954

Mentioned in SAL (#wikimedia-operations) [2019-07-25T17:33:46Z] <volans> running sudo cumin -s30 -b1 -m async 'A:wdqs-internal' 'run-puppet-agent -e "volans - T228122 - deploying gerrit/524954"' 'systemctl restart wdqs-blazegraph'

Volans added a subscriber: Volans.Jul 25 2019, 5:36 PM

errata corrige, I run the above with 'A:wdqs-internal and not P{wdqs1003.eqiad.wmnet}' instead to avoid to restart again 1003 that was already manually tested

Mentioned in SAL (#wikimedia-operations) [2019-07-25T17:44:18Z] <volans> sudo cumin -s30 -b1 -m async 'A:wdqs-all and not A:wdqs-internal and not P{wdqs1009.eqiad.wmnet}' 'run-puppet-agent -e "volans - T228122 - deploying gerrit/524954"' 'systemctl restart wdqs-blazegraph'