Page MenuHomePhabricator
Create Task
Maniphest T388134

Drop support for the full Wikidata graph from query.wikidata.org
Closed, ResolvedPublic
Actions

Description

Dropping support for the full graph and reallocating existing servers to serve the main graph instead. This should happen no sooner than April 10 to allow time for transition to query-legacy-full.wikidata.org for the services that needs it.

For a transition period, the full graph is available via query-legacy-full.wikidata.org (see T384422).

AC

  • query.wikidata.org serves the main graph
  • query-main.wikidata.org redirects (HTTP/301) to query.wikidata.org
  • query-scholary.wikidata.org serves the scholarly graph (as is already the case)
  • servers are allocated to the various clusters in a way that makes sense to share the load efficiently

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
[wip] wdqs: point query.wikidata.org to main graphoperations/deployment-chartsmaster+4 -4
wdqs-main: pool new hostsoperations/puppetproduction+9 -0
wdqs-main: shift over hosts from full graphoperations/puppetproduction+22 -25
wdqs: route query.wd.org to wdqs-mainoperations/puppetproduction+3 -3
wdqs-main: allow query.wikidata.org to hit mainoperations/puppetproduction+1 -0
wdqs: route query.wd.org to wdqs-mainoperations/puppetproduction+3 -3
wdqs-main: bring old internal hosts into serviceoperations/puppetproduction+12 -0
wdqs: add scap dsh targets for new wdqs-main hostsoperations/puppetproduction+6 -0
wdqs-main: switch old internal hosts to main graphoperations/puppetproduction+2 -2
Customize query in gerrit

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Stashbot added a comment.May 6 2025, 2:43 AM

Mentioned in SAL (#wikimedia-operations) [2025-05-06T02:43:45Z] <ryankemper@cumin2002> END (FAIL) - Cookbook sre.wdqs.data-transfer (exit_code=99) (T388134, bring new main graph hosts into service) xfer wikidata from wdqs1021.eqiad.wmnet -> wdqs1016.eqiad.wmnet, repooling source-only afterwards

Stashbot added a comment.May 6 2025, 2:49 AM

Mentioned in SAL (#wikimedia-operations) [2025-05-06T02:49:47Z] <ryankemper@cumin2002> START - Cookbook sre.wdqs.data-transfer (T388134, bring new main graph hosts into service) xfer wikidata from wdqs1021.eqiad.wmnet -> wdqs1016.eqiad.wmnet, repooling source-only afterwards

Stashbot added a comment.May 6 2025, 2:52 AM

Mentioned in SAL (#wikimedia-operations) [2025-05-06T02:52:14Z] <ryankemper@cumin2002> END (FAIL) - Cookbook sre.wdqs.data-transfer (exit_code=99) (T388134, bring new main graph hosts into service) xfer wikidata from wdqs1021.eqiad.wmnet -> wdqs1016.eqiad.wmnet, repooling source-only afterwards

Stashbot added a comment.May 6 2025, 3:00 AM

Mentioned in SAL (#wikimedia-operations) [2025-05-06T03:00:26Z] <ryankemper@cumin2002> START - Cookbook sre.wdqs.data-transfer (T388134, bring new main graph hosts into service) xfer wikidata_main from wdqs1021.eqiad.wmnet -> wdqs1016.eqiad.wmnet, repooling source-only afterwards

Stashbot added a comment.May 6 2025, 3:05 AM

Mentioned in SAL (#wikimedia-operations) [2025-05-06T03:05:07Z] <ryankemper@cumin2002> END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T388134, bring new main graph hosts into service) xfer wikidata_main from wdqs1021.eqiad.wmnet -> wdqs1016.eqiad.wmnet, repooling source-only afterwards

gerritbot added a comment.May 6 2025, 3:08 AM

Change #1142085 had a related patch set uploaded (by Ryan Kemper; author: Ryan Kemper):

[operations/puppet@production] wdqs: add scap dsh targets for new wdqs-main hosts

https://gerrit.wikimedia.org/r/1142085

gerritbot added a comment.May 6 2025, 3:09 AM

Change #1142085 merged by Ryan Kemper:

[operations/puppet@production] wdqs: add scap dsh targets for new wdqs-main hosts

https://gerrit.wikimedia.org/r/1142085

Stashbot added a comment.May 6 2025, 3:18 AM

Mentioned in SAL (#wikimedia-operations) [2025-05-06T03:18:35Z] <ryankemper@cumin2002> START - Cookbook sre.wdqs.data-transfer (T388134, bring new main graph hosts into service) xfer wikidata_main from wdqs1021.eqiad.wmnet -> wdqs1011.eqiad.wmnet, repooling source-only afterwards

Stashbot added a comment.May 6 2025, 3:25 AM

Mentioned in SAL (#wikimedia-operations) [2025-05-06T03:25:44Z] <ryankemper@cumin2002> END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T388134, bring new main graph hosts into service) xfer wikidata_main from wdqs1021.eqiad.wmnet -> wdqs1011.eqiad.wmnet, repooling source-only afterwards

Stashbot added a comment.May 6 2025, 3:27 AM

Mentioned in SAL (#wikimedia-operations) [2025-05-06T03:27:22Z] <ryankemper@cumin2002> START - Cookbook sre.wdqs.data-transfer (T388134, bring new main graph hosts into service) xfer wikidata_main from wdqs2021.codfw.wmnet -> wdqs2008.codfw.wmnet, repooling source-only afterwards

Stashbot added a comment.May 6 2025, 3:35 AM

Mentioned in SAL (#wikimedia-operations) [2025-05-06T03:35:39Z] <ryankemper@cumin2002> END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T388134, bring new main graph hosts into service) xfer wikidata_main from wdqs2021.codfw.wmnet -> wdqs2008.codfw.wmnet, repooling source-only afterwards

Stashbot added a comment.May 6 2025, 3:36 AM

Mentioned in SAL (#wikimedia-operations) [2025-05-06T03:36:02Z] <ryankemper@cumin2002> START - Cookbook sre.wdqs.data-transfer (T388134, bring new main graph hosts into service) xfer wikidata_main from wdqs2021.codfw.wmnet -> wdqs2014.codfw.wmnet, repooling source-only afterwards

Stashbot added a comment.May 6 2025, 3:44 AM

Mentioned in SAL (#wikimedia-operations) [2025-05-06T03:44:39Z] <ryankemper@cumin2002> END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T388134, bring new main graph hosts into service) xfer wikidata_main from wdqs2021.codfw.wmnet -> wdqs2014.codfw.wmnet, repooling source-only afterwards

Stashbot added a comment.May 6 2025, 3:45 AM

Mentioned in SAL (#wikimedia-operations) [2025-05-06T03:45:25Z] <ryankemper@cumin2002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 15:00:00 on wdqs[2008,2014-2015].codfw.wmnet,wdqs[1011,1016].eqiad.wmnet with reason: T388134

Stashbot added a comment.May 6 2025, 3:45 AM

Mentioned in SAL (#wikimedia-operations) [2025-05-06T03:45:38Z] <ryankemper@cumin2002> START - Cookbook sre.wdqs.data-transfer (T388134, bring new main graph hosts into service) xfer wikidata_main from wdqs2021.codfw.wmnet -> wdqs2015.codfw.wmnet, repooling source-only afterwards

Stashbot added a comment.May 6 2025, 3:52 AM

Mentioned in SAL (#wikimedia-operations) [2025-05-06T03:52:45Z] <ryankemper@cumin2002> END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T388134, bring new main graph hosts into service) xfer wikidata_main from wdqs2021.codfw.wmnet -> wdqs2015.codfw.wmnet, repooling source-only afterwards

Stashbot added a comment.May 6 2025, 5:31 PM

Mentioned in SAL (#wikimedia-operations) [2025-05-06T17:31:46Z] <ryankemper@cumin2002> START - Cookbook sre.wdqs.data-transfer (T388134, bring new main graph hosts into service) xfer wikidata_main from wdqs1021.eqiad.wmnet -> wdqs1017.eqiad.wmnet, repooling source-only afterwards

gerritbot added a comment.May 6 2025, 5:35 PM

Change #1141957 merged by Ryan Kemper:

[operations/puppet@production] wdqs-main: bring old internal hosts into service

https://gerrit.wikimedia.org/r/1141957

Stashbot added a comment.May 6 2025, 5:39 PM

Mentioned in SAL (#wikimedia-operations) [2025-05-06T17:39:24Z] <ryankemper@cumin2002> START - Cookbook sre.wdqs.data-transfer (T388134, bring new main graph hosts into service) xfer categories from wdqs2021.codfw.wmnet -> wdqs2008.codfw.wmnet, repooling source-only afterwards

Stashbot added a comment.May 6 2025, 5:40 PM

Mentioned in SAL (#wikimedia-operations) [2025-05-06T17:40:14Z] <ryankemper@cumin2002> START - Cookbook sre.wdqs.data-transfer (T388134, bring new main graph hosts into service) xfer categories from wdqs1021.eqiad.wmnet -> wdqs1011.eqiad.wmnet, repooling source-only afterwards

Stashbot added a comment.May 6 2025, 5:41 PM

Mentioned in SAL (#wikimedia-operations) [2025-05-06T17:41:08Z] <ryankemper@cumin2002> END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T388134, bring new main graph hosts into service) xfer wikidata_main from wdqs1021.eqiad.wmnet -> wdqs1017.eqiad.wmnet, repooling source-only afterwards

Stashbot added a comment.May 6 2025, 5:43 PM

Mentioned in SAL (#wikimedia-operations) [2025-05-06T17:43:57Z] <ryankemper@cumin2002> END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T388134, bring new main graph hosts into service) xfer categories from wdqs2021.codfw.wmnet -> wdqs2008.codfw.wmnet, repooling source-only afterwards

Stashbot added a comment.May 6 2025, 5:44 PM

Mentioned in SAL (#wikimedia-operations) [2025-05-06T17:44:41Z] <ryankemper@cumin2002> END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T388134, bring new main graph hosts into service) xfer categories from wdqs1021.eqiad.wmnet -> wdqs1011.eqiad.wmnet, repooling source-only afterwards

Stashbot added a comment.May 6 2025, 5:44 PM

Mentioned in SAL (#wikimedia-operations) [2025-05-06T17:44:44Z] <ryankemper@cumin2002> START - Cookbook sre.wdqs.data-transfer (T388134, bring new main graph hosts into service) xfer categories from wdqs2021.codfw.wmnet -> wdqs2014.codfw.wmnet, repooling source-only afterwards

Stashbot added a comment.May 6 2025, 5:44 PM

Mentioned in SAL (#wikimedia-operations) [2025-05-06T17:44:58Z] <ryankemper@cumin2002> START - Cookbook sre.wdqs.data-transfer (T388134, bring new main graph hosts into service) xfer categories from wdqs1021.eqiad.wmnet -> wdqs1016.eqiad.wmnet, repooling source-only afterwards

Stashbot added a comment.May 6 2025, 5:49 PM

Mentioned in SAL (#wikimedia-operations) [2025-05-06T17:49:04Z] <ryankemper@cumin2002> END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T388134, bring new main graph hosts into service) xfer categories from wdqs2021.codfw.wmnet -> wdqs2014.codfw.wmnet, repooling source-only afterwards

Stashbot added a comment.May 6 2025, 5:49 PM

Mentioned in SAL (#wikimedia-operations) [2025-05-06T17:49:36Z] <ryankemper@cumin2002> END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T388134, bring new main graph hosts into service) xfer categories from wdqs1021.eqiad.wmnet -> wdqs1016.eqiad.wmnet, repooling source-only afterwards

Stashbot added a comment.May 6 2025, 5:49 PM

Mentioned in SAL (#wikimedia-operations) [2025-05-06T17:49:38Z] <ryankemper@cumin2002> START - Cookbook sre.wdqs.data-transfer (T388134, bring new main graph hosts into service) xfer categories from wdqs2021.codfw.wmnet -> wdqs2015.codfw.wmnet, repooling source-only afterwards

Stashbot added a comment.May 6 2025, 5:49 PM

Mentioned in SAL (#wikimedia-operations) [2025-05-06T17:49:43Z] <ryankemper@cumin2002> START - Cookbook sre.wdqs.data-transfer (T388134, bring new main graph hosts into service) xfer categories from wdqs1021.eqiad.wmnet -> wdqs1017.eqiad.wmnet, repooling source-only afterwards

Stashbot added a comment.May 6 2025, 5:54 PM

Mentioned in SAL (#wikimedia-operations) [2025-05-06T17:54:09Z] <ryankemper@cumin2002> END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T388134, bring new main graph hosts into service) xfer categories from wdqs1021.eqiad.wmnet -> wdqs1017.eqiad.wmnet, repooling source-only afterwards

Stashbot added a comment.May 6 2025, 5:54 PM

Mentioned in SAL (#wikimedia-operations) [2025-05-06T17:54:12Z] <ryankemper@cumin2002> END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T388134, bring new main graph hosts into service) xfer categories from wdqs2021.codfw.wmnet -> wdqs2015.codfw.wmnet, repooling source-only afterwards

Stashbot added a comment.May 6 2025, 6:12 PM

Mentioned in SAL (#wikimedia-operations) [2025-05-06T18:12:09Z] <ryankemper@cumin2002> START - Cookbook sre.wdqs.data-transfer (T388134, bring new main graph hosts into service) xfer categories from wdqs1021.eqiad.wmnet -> wdqs1011.eqiad.wmnet w/ force delete existing files, repooling source-only afterwards

Stashbot added a comment.May 6 2025, 6:13 PM

Mentioned in SAL (#wikimedia-operations) [2025-05-06T18:13:30Z] <ryankemper@cumin2002> START - Cookbook sre.wdqs.data-transfer (T388134, bring new main graph hosts into service) xfer categories from wdqs2021.codfw.wmnet -> wdqs2008.codfw.wmnet w/ force delete existing files, repooling source-only afterwards

Stashbot added a comment.May 6 2025, 6:23 PM

Mentioned in SAL (#wikimedia-operations) [2025-05-06T18:23:35Z] <ryankemper@cumin2002> END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T388134, bring new main graph hosts into service) xfer categories from wdqs1021.eqiad.wmnet -> wdqs1011.eqiad.wmnet w/ force delete existing files, repooling source-only afterwards

Stashbot added a comment.May 6 2025, 6:24 PM

Mentioned in SAL (#wikimedia-operations) [2025-05-06T18:24:36Z] <ryankemper@cumin2002> END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T388134, bring new main graph hosts into service) xfer categories from wdqs2021.codfw.wmnet -> wdqs2008.codfw.wmnet w/ force delete existing files, repooling source-only afterwards

Stashbot added a comment.May 6 2025, 6:25 PM

Mentioned in SAL (#wikimedia-operations) [2025-05-06T18:25:17Z] <ryankemper@cumin2002> START - Cookbook sre.wdqs.data-transfer (T388134, bring new main graph hosts into service) xfer wikidata_main from wdqs1021.eqiad.wmnet -> wdqs1011.eqiad.wmnet w/ force delete existing files, repooling source-only afterwards

Stashbot added a comment.May 6 2025, 6:25 PM

Mentioned in SAL (#wikimedia-operations) [2025-05-06T18:25:43Z] <ryankemper@cumin2002> START - Cookbook sre.wdqs.data-transfer (T388134, bring new main graph hosts into service) xfer wikidata_main from wdqs2021.codfw.wmnet -> wdqs2014.codfw.wmnet w/ force delete existing files, repooling source-only afterwards

Stashbot added a comment.May 6 2025, 7:18 PM

Mentioned in SAL (#wikimedia-operations) [2025-05-06T19:18:39Z] <ryankemper@cumin2002> END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T388134, bring new main graph hosts into service) xfer wikidata_main from wdqs2021.codfw.wmnet -> wdqs2014.codfw.wmnet w/ force delete existing files, repooling source-only afterwards

Stashbot added a comment.May 6 2025, 7:20 PM

Mentioned in SAL (#wikimedia-operations) [2025-05-06T19:20:13Z] <ryankemper@cumin2002> START - Cookbook sre.wdqs.data-transfer (T388134, bring new main graph hosts into service) xfer wikidata_main from wdqs2021.codfw.wmnet -> wdqs2008.codfw.wmnet w/ force delete existing files, repooling source-only afterwards

Stashbot added a comment.May 6 2025, 7:21 PM

Mentioned in SAL (#wikimedia-operations) [2025-05-06T19:21:02Z] <ryankemper@cumin2002> START - Cookbook sre.wdqs.data-transfer (T388134, bring new main graph hosts into service) xfer wikidata_main from wdqs2014.codfw.wmnet -> wdqs2015.codfw.wmnet w/ force delete existing files, repooling neither afterwards

Stashbot added a comment.May 6 2025, 7:23 PM

Mentioned in SAL (#wikimedia-operations) [2025-05-06T19:23:36Z] <ryankemper@cumin2002> END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T388134, bring new main graph hosts into service) xfer wikidata_main from wdqs1021.eqiad.wmnet -> wdqs1011.eqiad.wmnet w/ force delete existing files, repooling source-only afterwards

Stashbot added a comment.May 6 2025, 7:25 PM

Mentioned in SAL (#wikimedia-operations) [2025-05-06T19:25:18Z] <ryankemper@cumin2002> START - Cookbook sre.wdqs.data-transfer (T388134, bring new main graph hosts into service) xfer wikidata_main from wdqs1021.eqiad.wmnet -> wdqs1016.eqiad.wmnet w/ force delete existing files, repooling source-only afterwards

Stashbot added a comment.May 6 2025, 7:25 PM

Mentioned in SAL (#wikimedia-operations) [2025-05-06T19:25:35Z] <ryankemper@cumin2002> START - Cookbook sre.wdqs.data-transfer (T388134, bring new main graph hosts into service) xfer wikidata_main from wdqs1011.eqiad.wmnet -> wdqs1017.eqiad.wmnet w/ force delete existing files, repooling neither afterwards

Stashbot added a comment.May 6 2025, 8:12 PM

Mentioned in SAL (#wikimedia-operations) [2025-05-06T20:12:50Z] <ryankemper@cumin2002> END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T388134, bring new main graph hosts into service) xfer wikidata_main from wdqs2014.codfw.wmnet -> wdqs2015.codfw.wmnet w/ force delete existing files, repooling neither afterwards

Stashbot added a comment.May 6 2025, 8:13 PM

Mentioned in SAL (#wikimedia-operations) [2025-05-06T20:13:35Z] <ryankemper@cumin2002> END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T388134, bring new main graph hosts into service) xfer wikidata_main from wdqs1011.eqiad.wmnet -> wdqs1017.eqiad.wmnet w/ force delete existing files, repooling neither afterwards

Stashbot added a comment.May 6 2025, 8:18 PM

Mentioned in SAL (#wikimedia-operations) [2025-05-06T20:18:03Z] <ryankemper@cumin2002> END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T388134, bring new main graph hosts into service) xfer wikidata_main from wdqs2021.codfw.wmnet -> wdqs2008.codfw.wmnet w/ force delete existing files, repooling source-only afterwards

Stashbot added a comment.May 6 2025, 8:24 PM

Mentioned in SAL (#wikimedia-operations) [2025-05-06T20:24:26Z] <ryankemper@cumin2002> END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T388134, bring new main graph hosts into service) xfer wikidata_main from wdqs1021.eqiad.wmnet -> wdqs1016.eqiad.wmnet w/ force delete existing files, repooling source-only afterwards

Stashbot added a comment.May 6 2025, 8:28 PM

Mentioned in SAL (#wikimedia-operations) [2025-05-06T20:28:25Z] <ryankemper@cumin2002> START - Cookbook sre.wdqs.data-transfer (T388134, bring new main graph hosts into service) xfer categories from wdqs2021.codfw.wmnet -> wdqs2014.codfw.wmnet w/ force delete existing files, repooling source-only afterwards

Stashbot added a comment.May 6 2025, 8:28 PM

Mentioned in SAL (#wikimedia-operations) [2025-05-06T20:28:31Z] <ryankemper@cumin2002> START - Cookbook sre.wdqs.data-transfer (T388134, bring new main graph hosts into service) xfer categories from wdqs1021.eqiad.wmnet -> wdqs1016.eqiad.wmnet w/ force delete existing files, repooling source-only afterwards

Stashbot added a comment.May 6 2025, 8:39 PM

Mentioned in SAL (#wikimedia-operations) [2025-05-06T20:39:30Z] <ryankemper@cumin2002> END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T388134, bring new main graph hosts into service) xfer categories from wdqs2021.codfw.wmnet -> wdqs2014.codfw.wmnet w/ force delete existing files, repooling source-only afterwards

Stashbot added a comment.May 6 2025, 8:39 PM

Mentioned in SAL (#wikimedia-operations) [2025-05-06T20:39:44Z] <ryankemper@cumin2002> END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T388134, bring new main graph hosts into service) xfer categories from wdqs1021.eqiad.wmnet -> wdqs1016.eqiad.wmnet w/ force delete existing files, repooling source-only afterwards

Stashbot added a comment.May 6 2025, 8:39 PM

Mentioned in SAL (#wikimedia-operations) [2025-05-06T20:39:56Z] <ryankemper@cumin2002> START - Cookbook sre.wdqs.data-transfer (T388134, bring new main graph hosts into service) xfer categories from wdqs2021.codfw.wmnet -> wdqs2015.codfw.wmnet w/ force delete existing files, repooling source-only afterwards

Stashbot added a comment.May 6 2025, 8:40 PM

Mentioned in SAL (#wikimedia-operations) [2025-05-06T20:40:09Z] <ryankemper@cumin2002> START - Cookbook sre.wdqs.data-transfer (T388134, bring new main graph hosts into service) xfer categories from wdqs1021.eqiad.wmnet -> wdqs1017.eqiad.wmnet w/ force delete existing files, repooling source-only afterwards

Stashbot added a comment.May 6 2025, 8:52 PM

Mentioned in SAL (#wikimedia-operations) [2025-05-06T20:52:21Z] <ryankemper@cumin2002> END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T388134, bring new main graph hosts into service) xfer categories from wdqs2021.codfw.wmnet -> wdqs2015.codfw.wmnet w/ force delete existing files, repooling source-only afterwards

Stashbot added a comment.May 6 2025, 8:52 PM

Mentioned in SAL (#wikimedia-operations) [2025-05-06T20:52:54Z] <ryankemper@cumin2002> END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T388134, bring new main graph hosts into service) xfer categories from wdqs1021.eqiad.wmnet -> wdqs1017.eqiad.wmnet w/ force delete existing files, repooling source-only afterwards

gerritbot added a comment.May 6 2025, 9:07 PM

Change #1139531 merged by Ryan Kemper:

[operations/puppet@production] wdqs: route query.wd.org to wdqs-main

https://gerrit.wikimedia.org/r/1139531

Stashbot added a comment.May 6 2025, 9:23 PM

Mentioned in SAL (#wikimedia-operations) [2025-05-06T21:23:28Z] <ryankemper> T388134 Cutover of query.wikidata.org to wdqs-main instead of wdqs is ongoing. We're seeing the expected drop in queries to the main cluster (https://grafana.wikimedia.org/d/000000489/wikidata-query-service?orgId=1&var-cluster_name=wdqs&from=1746565806937&to=1746566592047) but not seeing corresponding increase in wdqs-main yet

Stashbot added a comment.May 6 2025, 9:28 PM

Mentioned in SAL (#wikimedia-operations) [2025-05-06T21:28:16Z] <ryankemper> T388134 Seeing 502 errors; that explains why the drop in requests to wdqs-full is not matched by an increase to wdqs-main. Rolling back for now while we figure out what piece we're missing

gerritbot added a comment.May 7 2025, 9:49 PM

Change #1143194 had a related patch set uploaded (by Ryan Kemper; author: Ryan Kemper):

[operations/puppet@production] wdqs-main: allow query.wikidata.org to hit main

https://gerrit.wikimedia.org/r/1143194

gerritbot added a comment.May 8 2025, 1:30 PM

Change #1143194 merged by Bking:

[operations/puppet@production] wdqs-main: allow query.wikidata.org to hit main

https://gerrit.wikimedia.org/r/1143194

gerritbot added a comment.May 8 2025, 7:27 PM

Change #1143659 had a related patch set uploaded (by Ryan Kemper; author: Ryan Kemper):

[operations/puppet@production] wdqs: route query.wd.org to wdqs-main

https://gerrit.wikimedia.org/r/1143659

gerritbot added a comment.May 8 2025, 8:07 PM

Change #1143659 merged by Ryan Kemper:

[operations/puppet@production] wdqs: route query.wd.org to wdqs-main

https://gerrit.wikimedia.org/r/1143659

Stashbot added a comment.May 8 2025, 8:14 PM

Mentioned in SAL (#wikimedia-operations) [2025-05-08T20:14:57Z] <ryankemper> T388134 Beginning cutover of query.wikidata.org from wdqs to wdqs-main. Starting to see requests increase on wdqs-main (and decrease on wdqs) as expected. Rolling change to rest of cp text hosts. Traffic should be fully moved over in ~20 mins

gerritbot added a comment.May 8 2025, 8:27 PM

Change #1143670 had a related patch set uploaded (by Ryan Kemper; author: Ryan Kemper):

[operations/puppet@production] wdqs-main: shift over hosts from full graph

https://gerrit.wikimedia.org/r/1143670

gerritbot added a comment.May 8 2025, 9:07 PM

Change #1143670 merged by Bking:

[operations/puppet@production] wdqs-main: shift over hosts from full graph

https://gerrit.wikimedia.org/r/1143670

Stashbot added a comment.May 8 2025, 9:29 PM

Mentioned in SAL (#wikimedia-operations) [2025-05-08T21:29:50Z] <bking@cumin2002> START - Cookbook sre.wdqs.data-transfer (T388134, bring new main graph hosts into service) xfer wikidata_main from wdqs1011.eqiad.wmnet -> wdqs1012.eqiad.wmnet w/ force delete existing files, repooling source-only afterwards

Maintenance_bot removed a project: Patch-For-Review.May 8 2025, 9:30 PM
Stashbot added a comment.May 8 2025, 9:34 PM

Mentioned in SAL (#wikimedia-operations) [2025-05-08T21:34:03Z] <bking@cumin2002> START - Cookbook sre.wdqs.data-transfer (T388134, bring new main graph hosts into service) xfer wikidata_main from wdqs2014.codfw.wmnet -> wdqs2007.codfw.wmnet w/ force delete existing files, repooling source-only afterwards

ops-monitoring-bot added a comment.May 8 2025, 9:38 PM

Icinga downtime and Alertmanager silence (ID=6a4b9449-318a-4180-b66a-b5a140fed3a1) set by bking@cumin2002 for 6 days, 0:00:00 on 5 host(s) and their services with reason: bringing hosts online with a data transfer

wdqs[2007,2013].codfw.wmnet,wdqs[1012-1014].eqiad.wmnet
Stashbot added a comment.May 8 2025, 9:43 PM

Mentioned in SAL (#wikimedia-operations) [2025-05-08T21:43:20Z] <ryankemper> T388134 Cutover completed about an hour ago. Metrics look good; we're in the process of shifting over some of the old wdqs hosts to wdqs-main to increase capacity

Stashbot added a comment.May 8 2025, 10:27 PM

Mentioned in SAL (#wikimedia-operations) [2025-05-08T22:27:55Z] <bking@cumin2002> END (FAIL) - Cookbook sre.wdqs.data-transfer (exit_code=99) (T388134, bring new main graph hosts into service) xfer wikidata_main from wdqs2014.codfw.wmnet -> wdqs2007.codfw.wmnet w/ force delete existing files, repooling source-only afterwards

Stashbot added a comment.May 8 2025, 10:28 PM

Mentioned in SAL (#wikimedia-operations) [2025-05-08T22:28:51Z] <bking@cumin2002> END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T388134, bring new main graph hosts into service) xfer wikidata_main from wdqs1011.eqiad.wmnet -> wdqs1012.eqiad.wmnet w/ force delete existing files, repooling source-only afterwards

Stashbot added a comment.May 9 2025, 4:11 AM

Mentioned in SAL (#wikimedia-operations) [2025-05-09T04:11:50Z] <ryankemper@cumin2002> START - Cookbook sre.wdqs.data-transfer (T388134, bring new main graph hosts into service) xfer wikidata_main from wdqs2014.codfw.wmnet -> wdqs2007.codfw.wmnet w/ force delete existing files, repooling source-only afterwards

Stashbot added a comment.May 9 2025, 5:09 AM

Mentioned in SAL (#wikimedia-operations) [2025-05-09T05:09:56Z] <ryankemper@cumin2002> END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T388134, bring new main graph hosts into service) xfer wikidata_main from wdqs2014.codfw.wmnet -> wdqs2007.codfw.wmnet w/ force delete existing files, repooling source-only afterwards

Stashbot added a comment.May 9 2025, 5:12 AM

Mentioned in SAL (#wikimedia-operations) [2025-05-09T05:12:31Z] <ryankemper@cumin2002> START - Cookbook sre.wdqs.data-transfer (T388134, bring new main graph hosts into service) xfer wikidata_main from wdqs2007.codfw.wmnet -> wdqs2010.codfw.wmnet w/ force delete existing files, repooling neither afterwards

Stashbot added a comment.May 9 2025, 5:12 AM

Mentioned in SAL (#wikimedia-operations) [2025-05-09T05:12:42Z] <ryankemper@cumin2002> START - Cookbook sre.wdqs.data-transfer (T388134, bring new main graph hosts into service) xfer wikidata_main from wdqs1012.eqiad.wmnet -> wdqs1013.eqiad.wmnet w/ force delete existing files, repooling neither afterwards

Stashbot added a comment.May 9 2025, 6:07 AM

Mentioned in SAL (#wikimedia-operations) [2025-05-09T06:07:07Z] <ryankemper@cumin2002> END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T388134, bring new main graph hosts into service) xfer wikidata_main from wdqs1012.eqiad.wmnet -> wdqs1013.eqiad.wmnet w/ force delete existing files, repooling neither afterwards

Stashbot added a comment.May 9 2025, 6:10 AM

Mentioned in SAL (#wikimedia-operations) [2025-05-09T06:10:34Z] <ryankemper@cumin2002> END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T388134, bring new main graph hosts into service) xfer wikidata_main from wdqs2007.codfw.wmnet -> wdqs2010.codfw.wmnet w/ force delete existing files, repooling neither afterwards

Stashbot added a comment.May 9 2025, 6:25 AM

Mentioned in SAL (#wikimedia-operations) [2025-05-09T06:25:57Z] <ryankemper@cumin2002> START - Cookbook sre.wdqs.data-transfer (T388134, bring new main graph hosts into service) xfer wikidata_main from wdqs2007.codfw.wmnet -> wdqs2011.codfw.wmnet w/ force delete existing files, repooling neither afterwards

Stashbot added a comment.May 9 2025, 6:26 AM

Mentioned in SAL (#wikimedia-operations) [2025-05-09T06:26:03Z] <ryankemper@cumin2002> END (ERROR) - Cookbook sre.wdqs.data-transfer (exit_code=97) (T388134, bring new main graph hosts into service) xfer wikidata_main from wdqs2007.codfw.wmnet -> wdqs2011.codfw.wmnet w/ force delete existing files, repooling neither afterwards

Stashbot added a comment.May 9 2025, 6:26 AM

Mentioned in SAL (#wikimedia-operations) [2025-05-09T06:26:15Z] <ryankemper@cumin2002> START - Cookbook sre.wdqs.data-transfer (T388134, bring new main graph hosts into service) xfer wikidata_main from wdqs2007.codfw.wmnet -> wdqs2011.codfw.wmnet w/ force delete existing files, repooling neither afterwards

Stashbot added a comment.May 9 2025, 6:26 AM

Mentioned in SAL (#wikimedia-operations) [2025-05-09T06:26:26Z] <ryankemper@cumin2002> START - Cookbook sre.wdqs.data-transfer (T388134, bring new main graph hosts into service) xfer wikidata_main from wdqs2010.codfw.wmnet -> wdqs2012.codfw.wmnet w/ force delete existing files, repooling neither afterwards

Stashbot added a comment.May 9 2025, 6:27 AM

Mentioned in SAL (#wikimedia-operations) [2025-05-09T06:27:09Z] <ryankemper@cumin2002> START - Cookbook sre.wdqs.data-transfer (T388134, bring new main graph hosts into service) xfer wikidata_main from wdqs1012.eqiad.wmnet -> wdqs1014.eqiad.wmnet w/ force delete existing files, repooling neither afterwards

Stashbot added a comment.May 9 2025, 6:27 AM

Mentioned in SAL (#wikimedia-operations) [2025-05-09T06:27:32Z] <ryankemper@cumin2002> START - Cookbook sre.wdqs.data-transfer (T388134, bring new main graph hosts into service) xfer wikidata_main from wdqs1013.eqiad.wmnet -> wdqs1015.eqiad.wmnet w/ force delete existing files, repooling neither afterwards

Stashbot added a comment.May 9 2025, 7:22 AM

Mentioned in SAL (#wikimedia-operations) [2025-05-09T07:22:13Z] <ryankemper@cumin2002> END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T388134, bring new main graph hosts into service) xfer wikidata_main from wdqs1012.eqiad.wmnet -> wdqs1014.eqiad.wmnet w/ force delete existing files, repooling neither afterwards

Stashbot added a comment.May 9 2025, 7:22 AM

Mentioned in SAL (#wikimedia-operations) [2025-05-09T07:22:32Z] <ryankemper@cumin2002> END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T388134, bring new main graph hosts into service) xfer wikidata_main from wdqs1013.eqiad.wmnet -> wdqs1015.eqiad.wmnet w/ force delete existing files, repooling neither afterwards

Stashbot added a comment.May 9 2025, 7:22 AM

Mentioned in SAL (#wikimedia-operations) [2025-05-09T07:22:34Z] <ryankemper@cumin2002> END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T388134, bring new main graph hosts into service) xfer wikidata_main from wdqs2007.codfw.wmnet -> wdqs2011.codfw.wmnet w/ force delete existing files, repooling neither afterwards

Stashbot added a comment.May 9 2025, 7:23 AM

Mentioned in SAL (#wikimedia-operations) [2025-05-09T07:23:07Z] <ryankemper@cumin2002> END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T388134, bring new main graph hosts into service) xfer wikidata_main from wdqs2010.codfw.wmnet -> wdqs2012.codfw.wmnet w/ force delete existing files, repooling neither afterwards

Stashbot added a comment.May 9 2025, 7:24 AM

Mentioned in SAL (#wikimedia-operations) [2025-05-09T07:24:05Z] <ryankemper@cumin2002> START - Cookbook sre.wdqs.data-transfer (T388134, bring new main graph hosts into service) xfer wikidata_main from wdqs2010.codfw.wmnet -> wdqs2013.codfw.wmnet w/ force delete existing files, repooling neither afterwards

Stashbot added a comment.May 9 2025, 8:17 AM

Mentioned in SAL (#wikimedia-operations) [2025-05-09T08:17:07Z] <ryankemper@cumin2002> END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T388134, bring new main graph hosts into service) xfer wikidata_main from wdqs2010.codfw.wmnet -> wdqs2013.codfw.wmnet w/ force delete existing files, repooling neither afterwards

gerritbot added a comment.May 9 2025, 6:23 PM

Change #1143886 had a related patch set uploaded (by Ryan Kemper; author: Ryan Kemper):

[operations/puppet@production] wdqs-main: pull new hosts

https://gerrit.wikimedia.org/r/1143886

gerritbot added a project: Patch-For-Review.May 9 2025, 6:23 PM
gerritbot added a comment.May 9 2025, 6:28 PM

Change #1143886 merged by Ryan Kemper:

[operations/puppet@production] wdqs-main: pool new hosts

https://gerrit.wikimedia.org/r/1143886

gerritbot added a comment.May 9 2025, 8:46 PM

Change #1138935 abandoned by Ryan Kemper:

[operations/deployment-charts@master] [wip] wdqs: point query.wikidata.org to main graph

Reason:

unneeded

https://gerrit.wikimedia.org/r/1138935

RKemper closed this task as Resolved.May 14 2025, 7:11 PM
RKemper claimed this task.
RKemper updated the task description. (Show Details)
Gehel edited projects, added Data-Platform-SRE (2025.05.02 - 2025.05.23); removed Data-Platform-SRE.May 14 2025, 7:12 PM
Gehel moved this task from Backlog - project to Done on the Data-Platform-SRE (2025.05.02 - 2025.05.23) board.
Gehel moved this task from Done to Reported on the Data-Platform-SRE (2025.05.02 - 2025.05.23) board.May 16 2025, 9:23 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits