Page MenuHomePhabricator
Create Task
Maniphest T187766

Install / configure new WDQS servers
Closed, ResolvedPublic
Actions

Details

Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

Gehel triaged this task as High priority.Feb 20 2018, 9:50 AM
Gehel created this task.
Restricted Application added a project: Wikidata. · View Herald TranscriptFeb 20 2018, 9:50 AM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
Smalyshev moved this task from Incoming to Operations/SRE on the Wikidata-Query-Service board.Feb 22 2018, 4:48 AM
Smalyshev added subtasks: Unknown Object (Task), Unknown Object (Task), T187800: rack/setup/install wdqs200[4-6].Feb 22 2018, 6:33 PM
Smalyshev added a subtask: T182991: New WDQS clusters eqiad + codfw.
Lydia_Pintscher moved this task from incoming to monitoring on the Wikidata board.Feb 24 2018, 5:23 PM
Smalyshev mentioned this in T188045: wdqs1004 broken.Feb 27 2018, 6:26 PM
gerritbot subscribed.Mar 2 2018, 3:44 PM

Change 415872 had a related patch set uploaded (by Gehel; owner: Gehel):
[operations/puppet@production] [WIP] wdqs: configure the new internal cluster

https://gerrit.wikimedia.org/r/415872

gerritbot added a project: Patch-For-Review.Mar 2 2018, 3:44 PM
Gehel claimed this task.Mar 6 2018, 6:14 PM
Gehel moved this task from Backlog to In progress on the Discovery-Wikidata-Query-Service-Sprint board.
gerritbot added a comment.Mar 7 2018, 10:21 AM

Change 416921 had a related patch set uploaded (by Gehel; owner: Gehel):
[operations/puppet@production] wdqs: enable LDF server is now configurable

https://gerrit.wikimedia.org/r/416921

gerritbot added a comment.Mar 7 2018, 1:27 PM

Change 416921 merged by Gehel:
[operations/puppet@production] wdqs: enable LDF server is now configurable

https://gerrit.wikimedia.org/r/416921

gerritbot added a comment.Mar 7 2018, 3:02 PM

Change 416961 had a related patch set uploaded (by Gehel; owner: Gehel):
[operations/puppet@production] wdqs: configure new servers wdqs200[4-6]

https://gerrit.wikimedia.org/r/416961

gerritbot added a comment.Mar 8 2018, 8:25 AM

Change 417202 had a related patch set uploaded (by Gehel; owner: Gehel):
[operations/puppet@production] wdqs: use the raid10-gpt-srv-lvm-ext4 partman config for new wdqs nodes

https://gerrit.wikimedia.org/r/417202

gerritbot added a comment.Mar 8 2018, 1:26 PM

Change 415872 merged by Gehel:
[operations/puppet@production] wdqs: configure the new internal cluster

https://gerrit.wikimedia.org/r/415872

gerritbot added a comment.Mar 8 2018, 2:17 PM

Change 417202 merged by Gehel:
[operations/puppet@production] wdqs: use the raid10-gpt-srv-lvm-ext4 partman config for new wdqs nodes

https://gerrit.wikimedia.org/r/417202

ops-monitoring-bot subscribed.Mar 9 2018, 9:41 AM

Script wmf-auto-reimage was launched by gehel on neodymium.eqiad.wmnet for hosts:

['wdqs2004.codfw.wmnet', 'wdqs2005.codfw.wmnet', 'wdqs2006.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201803090941_gehel_15241.log.

gerritbot added a comment.Mar 9 2018, 9:42 AM

Change 416961 merged by Gehel:
[operations/puppet@production] wdqs: configure new servers wdqs200[4-6]

https://gerrit.wikimedia.org/r/416961

gerritbot added a comment.Mar 9 2018, 10:37 AM

Change 417782 had a related patch set uploaded (by Gehel; owner: Gehel):
[operations/puppet@production] wdqs: comment out wdqs_internal nodes from eqiad

https://gerrit.wikimedia.org/r/417782

gerritbot added a comment.Mar 9 2018, 10:37 AM

Change 417782 merged by Gehel:
[operations/puppet@production] wdqs: comment out wdqs_internal nodes from eqiad

https://gerrit.wikimedia.org/r/417782

ops-monitoring-bot added a comment.Mar 9 2018, 10:38 AM

Completed auto-reimage of hosts:

['wdqs2006.codfw.wmnet']

Of which those FAILED:

['wdqs2006.codfw.wmnet']
ops-monitoring-bot added a comment.Mar 9 2018, 10:55 AM

Script wmf-auto-reimage was launched by gehel on neodymium.eqiad.wmnet for hosts:

['wdqs2006.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201803091054_gehel_31807.log.

ops-monitoring-bot added a comment.Mar 9 2018, 11:20 AM

Completed auto-reimage of hosts:

['wdqs2006.codfw.wmnet']

and were ALL successful.

Gehel added a comment.Mar 12 2018, 2:29 PM

Initial data import is in progress on wdqs200[456] (note that wdqs2006 has issues with mgmt interface). Eqiad servers are not yet racked (T188432).

Gehel added a comment.Mar 13 2018, 7:58 PM

Data import for wdqs200[456] completed.

gerritbot added a comment.Mar 13 2018, 8:01 PM

Change 419264 had a related patch set uploaded (by Gehel; owner: Gehel):
[operations/puppet@production] wdqs: collect prometheus metrics for both wdqs clusters

https://gerrit.wikimedia.org/r/419264

gerritbot added a comment.Mar 14 2018, 3:03 PM

Change 419264 merged by Gehel:
[operations/puppet@production] wdqs: collect prometheus metrics for both wdqs clusters

https://gerrit.wikimedia.org/r/419264

gerritbot added a comment.Mar 15 2018, 10:47 AM

Change 419707 had a related patch set uploaded (by Gehel; owner: Gehel):
[operations/puppet@production] wdqs: add pigz package

https://gerrit.wikimedia.org/r/419707

gerritbot added a comment.Mar 15 2018, 5:52 PM

Change 419707 merged by Gehel:
[operations/puppet@production] wdqs: add pigz package

https://gerrit.wikimedia.org/r/419707

gerritbot added a comment.Apr 5 2018, 10:24 AM

Change 424260 had a related patch set uploaded (by Gehel; owner: Gehel):
[operations/puppet@production] wdqs: configure new servers wdqs100[6-8]

https://gerrit.wikimedia.org/r/424260

Gehel mentioned this in T188432: rack/setup/install wdqs100[6-8].Apr 5 2018, 10:28 AM
gerritbot added a comment.Apr 5 2018, 1:58 PM

Change 424260 merged by Gehel:
[operations/puppet@production] wdqs: configure new servers wdqs100[6-8]

https://gerrit.wikimedia.org/r/424260

Gehel mentioned this in T187800: rack/setup/install wdqs200[4-6].Apr 6 2018, 9:25 AM
Gehel closed subtask T187800: rack/setup/install wdqs200[4-6] as Resolved.
Gehel closed subtask Unknown Object (Task) as Resolved.
Gehel mentioned this in Unknown Object (Task).
Gehel closed subtask Unknown Object (Task) as Resolved.
gerritbot added a comment.Apr 6 2018, 1:13 PM

Change 424587 had a related patch set uploaded (by Gehel; owner: Gehel):
[operations/dns@master] wdqs: new wdqs-internal service

https://gerrit.wikimedia.org/r/424587

gerritbot added a comment.Apr 6 2018, 2:32 PM

Change 424599 had a related patch set uploaded (by Gehel; owner: Gehel):
[operations/puppet@production] wdqs: LVS and conftool configuration for new wdqs-internal service

https://gerrit.wikimedia.org/r/424599

gerritbot added a comment.Apr 9 2018, 1:35 PM

Change 425051 had a related patch set uploaded (by Gehel; owner: Gehel):
[operations/dns@master] wdqs-internal: new entry for service discovery

https://gerrit.wikimedia.org/r/425051

gerritbot added a comment.Apr 9 2018, 1:58 PM

Change 425051 abandoned by Gehel:
wdqs-internal: new entry for service discovery

Reason:
replaced by https://gerrit.wikimedia.org/r/#/c/424587/

https://gerrit.wikimedia.org/r/425051

gerritbot added a comment.Apr 10 2018, 2:25 PM

Change 425275 had a related patch set uploaded (by Gehel; owner: Gehel):
[wikidata/query/deploy@master] add new wdqs-internal cluster to scap targets

https://gerrit.wikimedia.org/r/425275

gerritbot added a comment.Apr 10 2018, 5:37 PM

Change 425275 merged by Smalyshev:
[wikidata/query/deploy@master] add new wdqs-internal cluster to scap targets

https://gerrit.wikimedia.org/r/425275

gerritbot added a comment.Apr 16 2018, 2:05 PM

Change 424587 merged by Gehel:
[operations/dns@master] wdqs: new wdqs-internal service

https://gerrit.wikimedia.org/r/424587

gerritbot added a comment.Apr 16 2018, 2:11 PM

Change 424599 merged by Gehel:
[operations/puppet@production] wdqs: LVS and conftool configuration for new wdqs-internal service

https://gerrit.wikimedia.org/r/424599

gerritbot added a comment.Apr 16 2018, 2:16 PM

Change 426926 had a related patch set uploaded (by Gehel; owner: Gehel):
[operations/puppet@production] wdqs-internal: new service discovery entry

https://gerrit.wikimedia.org/r/426926

gerritbot added a comment.Apr 16 2018, 2:18 PM

Change 426926 merged by Gehel:
[operations/puppet@production] wdqs-internal: new service discovery entry

https://gerrit.wikimedia.org/r/426926

Stashbot subscribed.Apr 16 2018, 2:25 PM

Mentioned in SAL (#wikimedia-operations) [2018-04-16T14:25:38Z] <vgutierrez> restarting pybal on lvs2006 - T187766

Stashbot added a comment.Apr 16 2018, 2:42 PM

Mentioned in SAL (#wikimedia-operations) [2018-04-16T14:42:00Z] <vgutierrez> restart pybal on lvs1006 - T187766

Stashbot added a comment.Apr 16 2018, 2:49 PM

Mentioned in SAL (#wikimedia-operations) [2018-04-16T14:49:41Z] <vgutierrez> restart pybal on lvs2003 - T187766

Stashbot added a comment.Apr 16 2018, 2:53 PM

Mentioned in SAL (#wikimedia-operations) [2018-04-16T14:53:13Z] <vgutierrez> restart pybal on lvs1003 - T187766

Gehel added a subscriber: Smalyshev.Apr 16 2018, 4:11 PM

The DC specific endpoints and the service discovery endpoint seem to work correctly:

  • curl -s wdqs-internal.svc.eqiad.wmnet/readiness-probe
  • curl -s wdqs-internal.svc.codfw.wmnet/readiness-probe
  • curl -s wdqs-internal.discovery.wmnet/readiness-probe (<- this is the endpoint to use for any internal client)

I'd like to have @Smalyshev to have a look and validate that this all looks correct before sending real traffic...

gerritbot added a comment.Apr 17 2018, 4:01 PM

Change 427160 had a related patch set uploaded (by Gehel; owner: Gehel):
[operations/puppet@production] wdqs: tune performance limits for the new wdqs-internal cluster

https://gerrit.wikimedia.org/r/427160

gerritbot added a comment.Apr 19 2018, 2:42 PM

Change 427160 merged by Gehel:
[operations/puppet@production] wdqs: tune performance limits for the new wdqs-internal cluster

https://gerrit.wikimedia.org/r/427160

Smalyshev closed subtask T182991: New WDQS clusters eqiad + codfw as Resolved.Apr 23 2018, 7:12 PM
Smalyshev moved this task from In progress to Done on the Discovery-Wikidata-Query-Service-Sprint board.Apr 26 2018, 4:24 AM
Smalyshev closed this task as Resolved.May 1 2018, 5:52 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL