Page MenuHomePhabricator
Create Task
Maniphest T376151

Cutover wdqs-internal to new split endpoints
Closed, ResolvedPublic
Actions

Description

Order of operations:

  • With the precondition that T376150 is complete, we should have a pool of codfw hosts that can serve existing wdqs-internal traffic. Eqiad needs to be depooled at the dns level and codfw needs to have any hosts that aren't the 5 chosen in T376150 depooled.
  • Take the old eqiad and codfw wdqs-internal hosts offline to reconfigure them as necessary for the new graph split
  • Bring hosts back online
  • Return the 5 codfw hosts in T376150 to wdqs full graph service.
  • Tear down the old wdqs-internal service completely, set up eqiad lvs for wdqs-internal-[main,scholarly]

Details

Other Assignee
bking
Related Changes in Gerrit:
SubjectRepoBranchLines +/-
wdqs: remove realserver includesoperations/puppetproduction+0 -3
Remove references to wdqs-internal listenter in fixturesoperations/deployment-chartsmaster+2 -8
wdqs-internal: remove lvs VIPoperations/dnsmaster+0 -2
wdqs-internal: remove old aliasoperations/puppetproduction+0 -1
wdqs-internal: rip out remaining logic/configoperations/puppetproduction+2 -106
wdqs-internal: remove service catalog entryoperations/puppetproduction+0 -49
wdqs-internal: remove from LBs and backend serversoperations/puppetproduction+1 -6
wdqs-internal: move back to lvs_setupoperations/puppetproduction+1 -1
wdqs-internal: remove disc recordsoperations/dnsmaster+0 -2
wdqs-internal: bring graph split into productionoperations/puppetproduction+2 -2
Customize query in gerrit

Related Objects
Search...

Event Timeline

RKemper created this task.Oct 1 2024, 2:52 PM
Gehel edited projects, added Data-Platform-SRE (2024.09.28 - 2024.10.18); removed Epic.Oct 9 2024, 7:32 PM
Gehel edited parent tasks, added: T374021: Make WikibaseQualityConstraints use split-graph query service; removed: T374453: [EPIC] Migration of traffic to new Split Endpoints for WDQS.Oct 9 2024, 7:52 PM
Gehel removed a project: Discovery-Search (Current work).Oct 14 2024, 3:44 PM
BTullis edited projects, added Data-Platform-SRE (2024.10.19 - 2024.11.08); removed Data-Platform-SRE (2024.09.28 - 2024.10.18).Oct 18 2024, 3:09 PM
dr0ptp4kt mentioned this in T374021: Make WikibaseQualityConstraints use split-graph query service.Oct 23 2024, 8:40 PM
Gehel edited parent tasks, added: T379023: Create WDQS split endpoints for internal traffic and reconfigure clients to use the new endpoints; removed: T374021: Make WikibaseQualityConstraints use split-graph query service.Nov 4 2024, 7:47 PM
Gehel edited projects, added Data-Platform-SRE (2024.11.09 - 2024.11.29); removed Data-Platform-SRE (2024.10.19 - 2024.11.08).Nov 8 2024, 9:48 AM
Gehel triaged this task as High priority.Nov 8 2024, 2:22 PM
RKemper updated the task description. (Show Details)Nov 22 2024, 12:03 AM
gerritbot added a comment.Nov 22 2024, 12:06 AM

Change #1094074 had a related patch set uploaded (by Ryan Kemper; author: Ryan Kemper):

[operations/puppet@production] wdqs-internal: bring graph split into production

https://gerrit.wikimedia.org/r/1094074

gerritbot added a project: Patch-For-Review.Nov 22 2024, 12:06 AM
Gehel edited projects, added Data-Platform-SRE (2024.11.30 - 2024.12.20); removed Data-Platform-SRE (2024.11.09 - 2024.11.29).Nov 29 2024, 1:23 PM
gerritbot added a comment.Dec 4 2024, 9:06 PM

Change #1094074 merged by Ryan Kemper:

[operations/puppet@production] wdqs-internal: bring graph split into production

https://gerrit.wikimedia.org/r/1094074

Maintenance_bot removed a project: Patch-For-Review.Dec 4 2024, 9:31 PM
Gehel edited projects, added Data-Platform-SRE (2025.01.11 - 2025.01.31); removed Data-Platform-SRE (2024.11.30 - 2024.12.20).Jan 10 2025, 10:39 AM
RKemper added a comment.EditedJan 14 2025, 4:00 PM
WDQS Internal Cutover Plan

Step 0: Do some curl tests from mediawiki to make sure they can talk to envoy of wdqs internal graph split
Step 1: Flip wikibaseconstraints in mediawiki-config https://phabricator.wikimedia.org/T374021

  • Q: How do we get change live? I think mediawiki-config changes are propagated without any restarts needed but we should verify

Step 2: Start switching over hosts from wdqs internal (non-graph-split) to wdqs internal graph split
Step 3: Tear down old pybal/lvs/etc config

Gehel edited projects, added Data-Platform-SRE (2025.02.10 - 2025.02.28); removed Data-Platform-SRE (2025.01.11 - 2025.01.31).Feb 11 2025, 10:05 AM
Gehel edited projects, added Data-Platform-SRE (2025.03.01 - 2025.03.21); removed Data-Platform-SRE (2025.02.10 - 2025.02.28).Feb 28 2025, 1:27 PM
Gehel edited projects, added Data-Platform-SRE (2025.03.22 - 2025.04.11); removed Data-Platform-SRE (2025.03.01 - 2025.03.21).Mar 21 2025, 9:57 AM
Gehel moved this task from Backlog - project to In Progress on the Data-Platform-SRE (2025.03.22 - 2025.04.11) board.Mar 21 2025, 10:47 AM
• Stevemunene moved this task from In Progress to Blocked/Waiting on the Data-Platform-SRE (2025.03.22 - 2025.04.11) board.Apr 2 2025, 9:42 AM
Gehel edited projects, added Data-Platform-SRE (2025.04.12 - 2025.05.02); removed Data-Platform-SRE (2025.03.22 - 2025.04.11).Apr 11 2025, 1:13 PM
Gehel moved this task from Backlog - project to Blocked/Waiting on the Data-Platform-SRE (2025.04.12 - 2025.05.02) board.
RKemper added a comment.Apr 15 2025, 2:53 PM

Remaining work:

  • Tear down LVS components for wdqs-internal
  • Convert old wdqs-internal hosts into wdqs-internal graph split and/or public wdqs graph split hosts
gerritbot added a comment.Apr 15 2025, 3:07 PM

Change #1136740 had a related patch set uploaded (by Ryan Kemper; author: Ryan Kemper):

[operations/dns@master] wdqs-internal: remove disc records

https://gerrit.wikimedia.org/r/1136740

gerritbot added a project: Patch-For-Review.Apr 15 2025, 3:07 PM
gerritbot added a comment.Apr 15 2025, 3:26 PM

Change #1136744 had a related patch set uploaded (by Ryan Kemper; author: Ryan Kemper):

[operations/puppet@production] wdqs-internal: move back to lvs_setup

https://gerrit.wikimedia.org/r/1136744

gerritbot added a comment.Apr 15 2025, 3:34 PM

Change #1136747 had a related patch set uploaded (by Ryan Kemper; author: Ryan Kemper):

[operations/puppet@production] wdqs-internal: remove from LBs and backend servers

https://gerrit.wikimedia.org/r/1136747

gerritbot added a comment.Apr 15 2025, 4:12 PM

Change #1136756 had a related patch set uploaded (by Ryan Kemper; author: Ryan Kemper):

[operations/puppet@production] wdqs-internal: remove service catalog entry

https://gerrit.wikimedia.org/r/1136756

gerritbot added a comment.Apr 15 2025, 4:12 PM

Change #1136757 had a related patch set uploaded (by Ryan Kemper; author: Ryan Kemper):

[operations/puppet@production] wdqs-internal: rip out remaining logic/config

https://gerrit.wikimedia.org/r/1136757

gerritbot added a comment.Apr 29 2025, 9:12 PM

Change #1139936 had a related patch set uploaded (by Ryan Kemper; author: Ryan Kemper):

[operations/dns@master] wdqs-internal: remove lvs VIP

https://gerrit.wikimedia.org/r/1139936

gerritbot added a comment.May 1 2025, 6:13 PM

Change #1136740 merged by Ryan Kemper:

[operations/dns@master] wdqs-internal: remove disc records

https://gerrit.wikimedia.org/r/1136740

gerritbot added a comment.May 1 2025, 6:24 PM

Change #1136744 merged by Ryan Kemper:

[operations/puppet@production] wdqs-internal: move back to lvs_setup

https://gerrit.wikimedia.org/r/1136744

Stashbot added a comment.May 1 2025, 6:26 PM

Mentioned in SAL (#wikimedia-operations) [2025-05-01T18:26:48Z] <ryankemper> T376151 (wdqs-internal lvs teardown) Merged https://gerrit.wikimedia.org/r/c/operations/puppet/+/1136744 to flip wdqs-internal service state to lvs_setup and running puppet across A:dnsbox

RKemper added a comment.May 1 2025, 6:37 PM

Steps for upcoming rolling restart of pybal here: https://phabricator.wikimedia.org/P75734

gerritbot added a comment.May 1 2025, 6:42 PM

Change #1136747 merged by Ryan Kemper:

[operations/puppet@production] wdqs-internal: remove from LBs and backend servers

https://gerrit.wikimedia.org/r/1136747

Stashbot added a comment.May 1 2025, 6:44 PM

Mentioned in SAL (#wikimedia-operations) [2025-05-01T18:44:54Z] <ryankemper> T376151 [wdqs-internal lvs teardown -> pybal rolling restart] ran puppet on O:Lvs::balancer after merging https://gerrit.wikimedia.org/r/c/operations/puppet/+/1136747

Stashbot added a comment.May 1 2025, 6:48 PM

Mentioned in SAL (#wikimedia-operations) [2025-05-01T18:48:38Z] <ryankemper> T376151 [wdqs-internal lvs teardown -> pybal rolling restart] Restarted pybal on A:lvs-secondary-eqiad, it only restarted on lvs1020 but for some reason lvs1013 doesn't have a pybal service running

Stashbot added a comment.May 1 2025, 6:55 PM

Mentioned in SAL (#wikimedia-operations) [2025-05-01T18:55:30Z] <ryankemper> T376151 [wdqs-internal lvs teardown -> pybal rolling restart] Restarted pybal on A:lvs-low-traffic-eqiad (lvs1019), waiting few mins before proceeding

Stashbot added a comment.May 1 2025, 6:58 PM

Mentioned in SAL (#wikimedia-operations) [2025-05-01T18:58:18Z] <ryankemper> T376151 [wdqs-internal lvs teardown -> pybal rolling restart] Restarted pybal on A:lvs-secondary-codfw (lvs2014), waiting 2 mins before proceeding

Stashbot added a comment.May 1 2025, 6:59 PM

Mentioned in SAL (#wikimedia-operations) [2025-05-01T18:59:44Z] <ryankemper> T376151 [wdqs-internal lvs teardown -> pybal rolling restart] Restarted pybal on A:lvs-low-traffic-codfw (lvs2013)

Stashbot added a comment.May 1 2025, 7:03 PM

Mentioned in SAL (#wikimedia-operations) [2025-05-01T19:03:05Z] <ryankemper> T376151 [wdqs-internal lvs teardown -> pybal rolling restart] ipvsadm --delete-service --tcp-service 10.2.1.41:80 on A:lvs-secondary-codfw OR A:lvs-low-traffic-codfw(lvs2013, lvs2014)

Stashbot added a comment.May 1 2025, 7:04 PM

Mentioned in SAL (#wikimedia-operations) [2025-05-01T19:04:21Z] <ryankemper> T376151 [wdqs-internal lvs teardown -> pybal rolling restart] ipvsadm --delete-service --tcp-service 10.2.2.41:80 on lvs1019 and lvs1020

Stashbot added a comment.May 1 2025, 7:09 PM

Mentioned in SAL (#wikimedia-operations) [2025-05-01T19:09:41Z] <ryankemper> T376151 [wdqs-internal lvs teardown -> pybal rolling restart] all IPVS diff check alerts have recovered, rolling restart complete

Stashbot added a comment.May 1 2025, 7:09 PM

Mentioned in SAL (#wikimedia-operations) [2025-05-01T19:09:57Z] <ryankemper> T376151 [wdqs-internal lvs teardown] running puppet across A:wdqs-internal now that pybal has been restarted

gerritbot added a comment.May 1 2025, 7:31 PM

Change #1140531 had a related patch set uploaded (by Ryan Kemper; author: Ryan Kemper):

[operations/puppet@production] wdqs: remove realserver includes

https://gerrit.wikimedia.org/r/1140531

gerritbot added a comment.May 1 2025, 7:36 PM

Change #1140531 merged by Ryan Kemper:

[operations/puppet@production] wdqs: remove realserver includes

https://gerrit.wikimedia.org/r/1140531

gerritbot added a comment.May 1 2025, 7:41 PM

Change #1136756 merged by Ryan Kemper:

[operations/puppet@production] wdqs-internal: remove service catalog entry

https://gerrit.wikimedia.org/r/1136756

gerritbot added a comment.May 1 2025, 7:44 PM

Change #1136757 merged by Ryan Kemper:

[operations/puppet@production] wdqs-internal: rip out remaining logic/config

https://gerrit.wikimedia.org/r/1136757

gerritbot added a comment.May 1 2025, 8:19 PM

Change #1140535 had a related patch set uploaded (by Ryan Kemper; author: Ryan Kemper):

[operations/puppet@production] wdqs-internal: remove old alias

https://gerrit.wikimedia.org/r/1140535

gerritbot added a comment.May 1 2025, 8:22 PM

Change #1140535 merged by Bking:

[operations/puppet@production] wdqs-internal: remove old alias

https://gerrit.wikimedia.org/r/1140535

gerritbot added a comment.May 1 2025, 8:46 PM

Change #1139936 merged by Ryan Kemper:

[operations/dns@master] wdqs-internal: remove lvs VIP

https://gerrit.wikimedia.org/r/1139936

Stashbot added a comment.May 1 2025, 8:50 PM

Mentioned in SAL (#wikimedia-operations) [2025-05-01T20:50:23Z] <ryankemper> T376151 [wdqs-internal lvs teardown] Surrendered 10.2.2.41/32 (eqiad wdqs-internal vip) and 10.2.1.41/32 (codfw wdqs-internal vip) from netbox interface

Stashbot added a comment.May 1 2025, 8:54 PM

Mentioned in SAL (#wikimedia-operations) [2025-05-01T20:54:53Z] <ryankemper> T376151 [wdqs-internal lvs teardown] sudo rm -fv /srv/config-master/pybal/eqiad/wdqs-internal && sudo rm -fv /srv/config-master/pybal/codfw/wdqs-internal on config-master[1,2]001

Stashbot added a comment.May 1 2025, 9:01 PM

Mentioned in SAL (#wikimedia-operations) [2025-05-01T21:01:00Z] <ryankemper> T376151 [wdqs-internal lvs teardown] sudo etcdctl -C https://conf1007.eqiad.wmnet:4001 --username root rmdir /conftool/v1/pools/eqiad/wdqs-internal/wdqs && sudo etcdctl -C https://conf1007.eqiad.wmnet:4001 --username root rmdir /conftool/v1/pools/eqiad/wdqs-internal/

Stashbot added a comment.May 1 2025, 9:01 PM

Mentioned in SAL (#wikimedia-operations) [2025-05-01T21:01:28Z] <ryankemper> T376151 [wdqs-internal lvs teardown] sudo etcdctl -C https://conf1007.eqiad.wmnet:4001 --username root rmdir /conftool/v1/pools/codfw/wdqs-internal/wdqs && sudo etcdctl -C https://conf1007.eqiad.wmnet:4001 --username root rmdir /conftool/v1/pools/codfw/wdqs-internal/

Stashbot added a comment.May 1 2025, 9:03 PM

Mentioned in SAL (#wikimedia-operations) [2025-05-01T21:03:01Z] <ryankemper> T376151 [wdqs-internal lvs teardown] Declaring this officially done. No more irc log spam from me today :)

Maintenance_bot removed a project: Patch-For-Review.May 1 2025, 9:30 PM
gerritbot added a comment.May 1 2025, 10:14 PM

Change #1140547 had a related patch set uploaded (by Scott French; author: Scott French):

[operations/deployment-charts@master] Remove references to wdqs-internal listenter in fixtures

https://gerrit.wikimedia.org/r/1140547

gerritbot added a project: Patch-For-Review.May 1 2025, 10:14 PM
gerritbot added a comment.May 1 2025, 10:22 PM

Change #1140547 merged by jenkins-bot:

[operations/deployment-charts@master] Remove references to wdqs-internal listenter in fixtures

https://gerrit.wikimedia.org/r/1140547

Maintenance_bot removed a project: Patch-For-Review.May 1 2025, 10:30 PM
Gehel edited projects, added Data-Platform-SRE (2025.05.02 - 2025.05.23); removed Data-Platform-SRE (2025.04.12 - 2025.05.02).May 5 2025, 12:48 PM
Gehel moved this task from Backlog - project to Blocked/Waiting on the Data-Platform-SRE (2025.05.02 - 2025.05.23) board.
ops-monitoring-bot added a comment.May 6 2025, 2:15 PM

Icinga downtime and Alertmanager silence (ID=67961f02-3a38-486d-8f1d-5d6bb2760fe4) set by bking@cumin2002 for 6 days, 0:00:00 on 1 host(s) and their services with reason: bringing host online after reimage

wdqs1017.eqiad.wmnet
bking subscribed.May 6 2025, 2:27 PM

Update on the wdqs1017 reimage that @RKemper started last night:

  • Reimage worked, but Puppet failed. I had to manually sign the cert and run puppet manually with puppet agent -tv from install-console.
  • Puppet "works", but cannot compile. Per our docs, I believe it needs a scap deploy . Ryan and/or myself will investigate further as time permits.
bking assigned this task to RKemper.May 6 2025, 2:28 PM
bking updated Other Assignee, added: bking.
bking moved this task from Blocked/Waiting to In Progress on the Data-Platform-SRE (2025.05.02 - 2025.05.23) board.
RKemper moved this task from In Progress to Done on the Data-Platform-SRE (2025.05.02 - 2025.05.23) board.May 7 2025, 6:14 AM
Gehel moved this task from Done to Reported on the Data-Platform-SRE (2025.05.02 - 2025.05.23) board.May 9 2025, 8:56 AM
Gehel closed this task as Resolved.May 9 2025, 9:10 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits