Page MenuHomePhabricator
Create Task
Maniphest T385896

Deploy wdqs-categories on wdqs-main/wdqs-internal-main hosts
Closed, ResolvedPublic
Actions

Description

As discussed in T375520, we need a new home for the wdqs-categories service. While it will eventually live in Kubernetes, we have a deadline to retire the legacy wdqs "full graph" servers by the end of March 2025 and we will not be ready for Kubernetes by then.

As such, wdqs-categories will live on the wdqs-main and wdqs-internal-main tier hosts until the Kubernetes migration.

Creating this ticket to:

  • update code (puppet and mediawiki config)
  • apply Puppet code/load categories graph on all hosts
  • anything else that may be necessary

Details

Other Assignee
RKemper
Related Changes in Gerrit:
SubjectAuthorRepoBranchLines +/-
wdqs categories: switch to internal-mainRyan Kemperoperations/mediawiki-configmaster+1 -1
wdqs-categories: remove extraneous wgCirrusSearchCategoryEndpoint valueBkingoperations/mediawiki-configmaster+0 -5
wdqs-main: Start hosting wdqs-categories serviceBkingoperations/puppetproduction+10 -0
Customize query in gerrit

Related Objects
Search...

Event Timeline

bking created this task.Feb 7 2025, 5:10 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 7 2025, 5:10 PM
gerritbot added a comment.Feb 7 2025, 5:10 PM

Change #1118124 had a related patch set uploaded (by Bking; author: Bking):

[operations/puppet@production] wdqs-main: Start hosting wdqs-categories service

https://gerrit.wikimedia.org/r/1118124

gerritbot added a project: Patch-For-Review.Feb 7 2025, 5:10 PM

Change #1118124 merged by Bking:

[operations/puppet@production] wdqs-main: Start hosting wdqs-categories service

https://gerrit.wikimedia.org/r/1118124

bking added a parent task: T375520: EPIC: WDQS categories migration.Feb 7 2025, 5:15 PM
bking mentioned this in T380608: Address categories migration for internal graph split endpoints.
Maintenance_bot removed a project: Patch-For-Review.Feb 7 2025, 5:30 PM
bking changed the task status from Open to In Progress.Feb 7 2025, 8:26 PM
bking triaged this task as High priority.
bking updated Other Assignee, added: RKemper.
bking added subscribers: dcausse, RKemper.
bking mentioned this in T385909: Discuss/potentially change wdqs-categories alert routing.Feb 7 2025, 9:03 PM
Gehel edited projects, added: Data-Platform-SRE (2025.02.10 - 2025.02.28); removed: Data-Platform-SRE (2025.01.11 - 2025.01.31).Feb 11 2025, 10:04 AM
bking moved this task from Backlog - project to In Progress on the Data-Platform-SRE (2025.02.10 - 2025.02.28) board.Feb 11 2025, 3:06 PM
bking mentioned this in T380934: decommission wdqs200[7-8].Feb 11 2025, 3:41 PM
bking updated the task description. (Show Details)Feb 11 2025, 6:47 PM
bking mentioned this in T374967: wdqs-categories migration: decide where to migrate.Feb 14 2025, 2:26 PM
gerritbot added a comment.Feb 24 2025, 2:46 PM

Change #1122151 had a related patch set uploaded (by Bking; author: Bking):

[operations/mediawiki-config@master] wdqs-categories: use new split graph hosts (wdqs-main) for categories

https://gerrit.wikimedia.org/r/1122151

gerritbot added a project: Patch-For-Review.Feb 24 2025, 2:46 PM
bking updated the task description. (Show Details)Feb 24 2025, 3:37 PM
Gehel edited projects, added: Data-Platform-SRE (2025.03.01 - 2025.03.21); removed: Data-Platform-SRE (2025.02.10 - 2025.02.28).Feb 28 2025, 1:27 PM
Gehel moved this task from Backlog - project to In Progress on the Data-Platform-SRE (2025.03.01 - 2025.03.21) board.
gerritbot added a comment.Mar 4 2025, 10:38 PM

Change #1124535 had a related patch set uploaded (by Ryan Kemper; author: Ryan Kemper):

[operations/mediawiki-config@master] wdqs categories: switch to internal-main

https://gerrit.wikimedia.org/r/1124535

Stashbot added a comment.Mar 17 2025, 7:04 PM

Mentioned in SAL (#wikimedia-operations) [2025-03-17T19:04:06Z] <ryankemper@cumin2002> START - Cookbook sre.wdqs.data-transfer (T385896, xfer categories jnl) xfer categories from wdqs1021.eqiad.wmnet -> wdqs1025.eqiad.wmnet, repooling both afterwards

Stashbot added a comment.Mar 17 2025, 7:08 PM

Mentioned in SAL (#wikimedia-operations) [2025-03-17T19:08:48Z] <ryankemper@cumin2002> END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T385896, xfer categories jnl) xfer categories from wdqs1021.eqiad.wmnet -> wdqs1025.eqiad.wmnet, repooling both afterwards

RKemper added a comment.Mar 17 2025, 7:14 PM

Categories is running properly on all the wdqs*main* hosts, so we should be good to merge https://gerrit.wikimedia.org/r/1124535 when we're ready.

Gehel edited projects, added: Data-Platform-SRE (2025.03.22 - 2025.04.11); removed: Data-Platform-SRE (2025.03.01 - 2025.03.21).Mar 21 2025, 9:57 AM
Gehel moved this task from Backlog - project to In Progress on the Data-Platform-SRE (2025.03.22 - 2025.04.11) board.
gerritbot added a comment.Mar 25 2025, 8:24 PM

Change #1122151 merged by Ryan Kemper:

[operations/mediawiki-config@master] wdqs-categories: remove extraneous wgCirrusSearchCategoryEndpoint value

https://gerrit.wikimedia.org/r/1122151

gerritbot added a comment.Mar 25 2025, 8:25 PM

Change #1124535 merged by jenkins-bot:

[operations/mediawiki-config@master] wdqs categories: switch to internal-main

https://gerrit.wikimedia.org/r/1124535

Stashbot added a comment.Mar 25 2025, 8:26 PM

Mentioned in SAL (#wikimedia-operations) [2025-03-25T20:26:26Z] <ryankemper@deploy1003> Started scap sync-world: Backport for [[gerrit:1124535|wdqs categories: switch to internal-main (T375520 T385896 T337013)]]

Stashbot mentioned this in T375520: EPIC: WDQS categories migration.Mar 25 2025, 8:26 PM
Stashbot mentioned this in T337013: [Epic] Splitting the graph in WDQS.
Maintenance_bot removed a project: Patch-For-Review.Mar 25 2025, 8:29 PM
Stashbot added a comment.Mar 25 2025, 8:33 PM

Mentioned in SAL (#wikimedia-operations) [2025-03-25T20:33:11Z] <ryankemper@deploy1003> ryankemper: Backport for [[gerrit:1124535|wdqs categories: switch to internal-main (T375520 T385896 T337013)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)

Stashbot added a comment.Mar 25 2025, 8:40 PM

Mentioned in SAL (#wikimedia-operations) [2025-03-25T20:40:21Z] <ryankemper> T385896 Got successful deepcat search of deepcat:"musicals" on en.wikipedia.org with X-Wikimedia-Debug:backend=k8s-mwdebug; rolling out change fully now

Stashbot added a comment.Mar 25 2025, 8:48 PM

Mentioned in SAL (#wikimedia-operations) [2025-03-25T20:48:06Z] <ryankemper@deploy1003> Finished scap sync-world: Backport for [[gerrit:1124535|wdqs categories: switch to internal-main (T375520 T385896 T337013)]] (duration: 21m 40s)

bking closed this task as Resolved.Mar 31 2025, 1:49 PM
bking moved this task from In Progress to Done on the Data-Platform-SRE (2025.03.22 - 2025.04.11) board.

wdqs-categories is deployed on the graph split hosts, and Mediawiki has been updated to point to the graph split hosts as well, so I'm closing this out.

Gehel moved this task from Done to Reported on the Data-Platform-SRE (2025.03.22 - 2025.04.11) board.Apr 4 2025, 7:54 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits