Page MenuHomePhabricator

Switch wdqs1003 with one of the internal wdqs cluster
Closed, ResolvedPublic

Description

Since wdqs1003 is acting differently from other servers in the cluster (T200563), let's try to switch it with one of the internal cluster. This might help us validate how much of what we see is related to the kind of load we have on the public cluster, and how much is related to the number 3.

Event Timeline

Gehel triaged this task as High priority.Oct 25 2018, 2:15 PM
Gehel created this task.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 25 2018, 2:15 PM
Smalyshev renamed this task from https://phabricator.wikimedia.org/T200563Switch wdqs1003 with one of the internal wdqs cluster to Switch wdqs1003 with one of the internal wdqs cluster.Oct 25 2018, 3:28 PM

Change 469649 had a related patch set uploaded (by Gehel; owner: Gehel):
[operations/puppet@production] wdqs: switch wdqs1003 and wdqs1006 from public vs internal clusters

https://gerrit.wikimedia.org/r/469649

ema added a subscriber: ema.EditedOct 26 2018, 7:23 AM

You can see which hosts are part of which service using the "service" dropdown on the PyBal Grafana dashboard. The primary LVS serving wdqs and wdqs-internal is lvs1016.eqiad.wmnet, the secondary is lvs1006.wikimedia.org.

Change 469649 abandoned by Gehel:
wdqs: switch wdqs1003 and wdqs1006 from public vs internal clusters

Reason:
superseeded

https://gerrit.wikimedia.org/r/469649

Change 469685 had a related patch set uploaded (by Gehel; owner: Gehel):
[operations/puppet@production] wdqs: remove wdqs1006 from internal cluster

https://gerrit.wikimedia.org/r/469685

Change 469686 had a related patch set uploaded (by Gehel; owner: Gehel):
[operations/puppet@production] wdqs: add wdqs1006 to public cluster

https://gerrit.wikimedia.org/r/469686

Change 469687 had a related patch set uploaded (by Gehel; owner: Gehel):
[operations/puppet@production] wdqs: remove wdqs1003 from public cluster

https://gerrit.wikimedia.org/r/469687

Change 469688 had a related patch set uploaded (by Gehel; owner: Gehel):
[operations/puppet@production] wdqs: add wdqs1003 to internal cluster

https://gerrit.wikimedia.org/r/469688

Mentioned in SAL (#wikimedia-operations) [2018-10-29T10:03:20Z] <gehel> starting to switch wdqs1003 and wdqs1006 - T207947

Change 469685 merged by Gehel:
[operations/puppet@production] wdqs: remove wdqs1006 from internal cluster

https://gerrit.wikimedia.org/r/469685

Change 469686 merged by Gehel:
[operations/puppet@production] wdqs: add wdqs1006 to public cluster

https://gerrit.wikimedia.org/r/469686

Change 469687 merged by Gehel:
[operations/puppet@production] wdqs: remove wdqs1003 from public cluster

https://gerrit.wikimedia.org/r/469687

Change 469688 merged by Gehel:
[operations/puppet@production] wdqs: add wdqs1003 to internal cluster

https://gerrit.wikimedia.org/r/469688

Mentioned in SAL (#wikimedia-operations) [2018-10-29T10:22:55Z] <gehel> switch wdqs1003 and wdqs1006 completed, wdqs1003 still depooled to catch up on update lag - T207947

Addshore moved this task from incoming to monitoring on the Wikidata board.Oct 31 2018, 5:08 PM
Smalyshev closed this task as Resolved.Nov 6 2018, 6:26 PM
Smalyshev claimed this task.