This task is for reloading WDQS database from scratch. The plan for hosts is as follows:
Only one machine per cluster (public or internal) should be loaded at the same time, but I think it's ok to load one public and one internal one in parallel.
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | Gehel | T222404 Tasks requiring Blazegraph reload | |||
| Resolved | Gehel | T228122 DB reload for WDQS |
Change 523266 had a related patch set uploaded (by Smalyshev; owner: Gehel):
[operations/puppet@production] wdqs: introduced tuned journal options to wdqs2001.
Hmm I am seeing some missing items in wdqs1010, so let's not update for now until I find out what's up.
@Gehel Never mind, this was me misconfiguring the query to use wrong namespace :\ The data is fine, so we can proceed.
Mentioned in SAL (#wikimedia-operations) [2019-07-16T18:54:56Z] <gehel> data copy from wdqs2004 to wdqs2001 - T228122
Change 523266 merged by Gehel:
[operations/puppet@production] wdqs: introduced tuned journal options to wdqs2001.
Change 523866 had a related patch set uploaded (by Gehel; owner: Gehel):
[operations/puppet@production] wdqs: introduced tuned journal options to wdqs1003.
Change 523867 had a related patch set uploaded (by Gehel; owner: Gehel):
[operations/puppet@production] wdqs: introduced tuned journal options to wdqs1004.
Change 523868 had a related patch set uploaded (by Gehel; owner: Gehel):
[operations/puppet@production] wdqs: introduced tuned journal options to wdqs2005.
Change 523869 had a related patch set uploaded (by Gehel; owner: Gehel):
[operations/puppet@production] wdqs: introduced tuned journal options to wdqs2006.
Change 523870 had a related patch set uploaded (by Gehel; owner: Gehel):
[operations/puppet@production] wdqs: introduced tuned journal options to wdqs1007.
Change 523871 had a related patch set uploaded (by Gehel; owner: Gehel):
[operations/puppet@production] wdqs: introduced tuned journal options to wdqs1008.
Change 523872 had a related patch set uploaded (by Gehel; owner: Gehel):
[operations/puppet@production] wdqs: introduced tuned journal options to wdqs2002.
Change 523873 had a related patch set uploaded (by Gehel; owner: Gehel):
[operations/puppet@production] wdqs: introduced tuned journal options to wdqs2003.
Change 523874 had a related patch set uploaded (by Gehel; owner: Gehel):
[operations/puppet@production] wdqs: introduced tuned journal options to wdqs1005.
Change 523875 had a related patch set uploaded (by Gehel; owner: Gehel):
[operations/puppet@production] wdqs: introduced tuned journal options to wdqs1006.
Change 523866 merged by Gehel:
[operations/puppet@production] wdqs: introduced tuned journal options to wdqs1003.
Change 523867 merged by Gehel:
[operations/puppet@production] wdqs: introduced tuned journal options to wdqs1004.
Change 523868 merged by Gehel:
[operations/puppet@production] wdqs: introduced tuned journal options to wdqs2005.
Change 523869 merged by Gehel:
[operations/puppet@production] wdqs: introduced tuned journal options to wdqs2006.
Change 523870 merged by Gehel:
[operations/puppet@production] wdqs: introduced tuned journal options to wdqs1007.
Change 523871 merged by Gehel:
[operations/puppet@production] wdqs: introduced tuned journal options to wdqs1008.
Change 523872 merged by Gehel:
[operations/puppet@production] wdqs: introduced tuned journal options to wdqs2002.
Change 523873 merged by Gehel:
[operations/puppet@production] wdqs: introduced tuned journal options to wdqs2003.
Icinga says for wdqs2004: "PYBAL CRITICAL - CRITICAL - wdqs-internal_80: Servers wdqs2004.codfw.wmnet are marked down but pooled"
and for wdqs1010: "PROCS CRITICAL: 0 processes with UID = 499 (blazegraph), regex args '^java .* --port 9999 .* blazegraph-service-.*war'"
Mentioned in SAL (#wikimedia-operations) [2019-07-19T08:22:57Z] <gehel> repooling wdqs2003 - T228122
Mentioned in SAL (#wikimedia-operations) [2019-07-19T08:24:02Z] <gehel> repooling wdqs2004 - T228122
Change 523874 merged by Gehel:
[operations/puppet@production] wdqs: introduced tuned journal options to wdqs1005.
Change 523875 merged by Gehel:
[operations/puppet@production] wdqs: introduced tuned journal options to wdqs1006.
Change 524954 had a related patch set uploaded (by Smalyshev; owner: Smalyshev):
[operations/puppet@production] Update WDQS standard settings to new DB settings
Mentioned in SAL (#wikimedia-operations) [2019-07-25T17:18:15Z] <volans> disabled puppet on A:wdqs-all, deploying gerrit/524954 - T228122
Change 524954 merged by Volans:
[operations/puppet@production] Update WDQS standard settings to new DB settings
Mentioned in SAL (#wikimedia-operations) [2019-07-25T17:33:46Z] <volans> running sudo cumin -s30 -b1 -m async 'A:wdqs-internal' 'run-puppet-agent -e "volans - T228122 - deploying gerrit/524954"' 'systemctl restart wdqs-blazegraph'
errata corrige, I run the above with 'A:wdqs-internal and not P{wdqs1003.eqiad.wmnet}' instead to avoid to restart again 1003 that was already manually tested
Mentioned in SAL (#wikimedia-operations) [2019-07-25T17:44:18Z] <volans> sudo cumin -s30 -b1 -m async 'A:wdqs-all and not A:wdqs-internal and not P{wdqs1009.eqiad.wmnet}' 'run-puppet-agent -e "volans - T228122 - deploying gerrit/524954"' 'systemctl restart wdqs-blazegraph'