Page MenuHomePhabricator
Create Task
Maniphest T395772

Teardown lvs for wdqs public pool
Closed, ResolvedPublic
Actions

Description

With the graph split cutover being completed, the next step is to tear down the old wdqs public lvs pool and free up the remaining hosts.

Coordinate with traffic team for the lvs teardown

Details

Other Assignee
• Stevemunene
Related Changes in Gerrit:
SubjectRepoBranchLines +/-
wdqs: shift old full graph hosts to new rolesoperations/puppetproduction+6 -12
wdqs: these hosts no longer in wdqs-publicoperations/puppetproduction+0 -5
wdqs: point to wdqs-main svc for maxlagoperations/puppetproduction+1 -1
wdqs: (steps 5,6) => final removaloperations/puppetproduction+0 -117
wdqs: remove ipipoperations/puppetproduction+0 -1
wdqs: shift old wdqs-public hosts to testoperations/puppetproduction+15 -9
wdqs: (step 4) remove from LBs and wdqs backendsoperations/puppetproduction+3 -9
wdqs: (step 3) shift service state to lvs_setupoperations/puppetproduction+3 -3
wdqs: (step 2) remove wdqs discovery dns recordsoperations/dnsmaster+0 -4
Customize query in gerrit

Related Objects
Search...

Event Timeline

RKemper created this task.Jun 2 2025, 7:03 AM
RKemper edited projects, added: Data-Platform-SRE (2025.05.24 - 2025.06.13); removed: Discovery-Search.
RKemper updated the task description. (Show Details)
RKemper renamed this task from Teardown lvs for wdqs public to Teardown lvs for wdqs public pool.Jun 2 2025, 7:06 AM
Gehel edited projects, added: Data-Platform-SRE (2025.06.13 - 2025.07.04); removed: Data-Platform-SRE (2025.05.24 - 2025.06.13).Jun 13 2025, 8:49 AM
Stashbot added a comment.Jun 18 2025, 6:56 PM

Mentioned in SAL (#wikimedia-operations) [2025-06-18T18:56:18Z] <ryankemper@cumin1003> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5 days, 0:00:00 on 6 hosts with reason: T395772 hosts not serving production traffic

Gehel triaged this task as Medium priority.Jun 20 2025, 8:06 AM
Gehel moved this task from Incoming to Operations/SRE on the Wikidata-Query-Service board.
BTullis edited projects, added: Data-Platform-SRE (2025.07.05 - 2025.07.25); removed: Data-Platform-SRE (2025.06.13 - 2025.07.04).Jul 4 2025, 3:59 PM
BTullis moved this task from Backlog - project to In Progress on the Data-Platform-SRE (2025.07.05 - 2025.07.25) board.Jul 4 2025, 4:20 PM
BTullis edited projects, added: Data-Platform-SRE (2025.07.26 - 2025.08.15); removed: Data-Platform-SRE (2025.07.05 - 2025.07.25).Jul 28 2025, 4:47 PM
BTullis moved this task from Backlog - project to In Progress on the Data-Platform-SRE (2025.07.26 - 2025.08.15) board.
BTullis edited projects, added: Data-Platform-SRE (2025.08.16 - 2025.09.05); removed: Data-Platform-SRE (2025.07.26 - 2025.08.15).Aug 20 2025, 10:30 AM
BTullis moved this task from Backlog - project to In Progress on the Data-Platform-SRE (2025.08.16 - 2025.09.05) board.
gerritbot added a comment.Aug 29 2025, 4:02 AM

Change #1182975 had a related patch set uploaded (by Ryan Kemper; author: Ryan Kemper):

[operations/puppet@production] wdqs: (step 3) shift service state to lvs_setup

https://gerrit.wikimedia.org/r/1182975

gerritbot added a project: Patch-For-Review.Aug 29 2025, 4:02 AM

Change #1182976 had a related patch set uploaded (by Ryan Kemper; author: Ryan Kemper):

[operations/dns@master] wdqs: (step 2) remove wdqs discovery dns records

https://gerrit.wikimedia.org/r/1182976

gerritbot added a comment.Aug 29 2025, 4:15 AM

Change #1182977 had a related patch set uploaded (by Ryan Kemper; author: Ryan Kemper):

[operations/puppet@production] wdqs: (step 4) remove from LBs and wdqs backends

https://gerrit.wikimedia.org/r/1182977

gerritbot added a comment.Aug 29 2025, 4:15 AM

Change #1182978 had a related patch set uploaded (by Ryan Kemper; author: Ryan Kemper):

[operations/puppet@production] wdqs: (steps 5,6) => final removal

https://gerrit.wikimedia.org/r/1182978

Gehel edited projects, added: Data-Platform-SRE (2025.09.05 - 2025.09.26); removed: Data-Platform-SRE (2025.08.16 - 2025.09.05).Sep 5 2025, 2:24 PM
Gehel moved this task from Backlog - project to In Progress on the Data-Platform-SRE (2025.09.05 - 2025.09.26) board.
• Stevemunene updated Other Assignee, added: • Stevemunene.Sep 9 2025, 9:42 AM
Gehel added a project: Essential-Work.Sep 11 2025, 12:12 PM
BCornwall subscribed.Sep 15 2025, 4:47 PM
BCornwall added a comment.Sep 16 2025, 3:36 PM

Hi, @RKemper! Would you like to schedule some time to go through this process?

gerritbot added a comment.Sep 18 2025, 8:57 PM

Change #1182976 merged by BCornwall:

[operations/dns@master] wdqs: (step 2) remove wdqs discovery dns records

https://gerrit.wikimedia.org/r/1182976

gerritbot added a comment.Sep 18 2025, 9:08 PM

Change #1182975 merged by BCornwall:

[operations/puppet@production] wdqs: (step 3) shift service state to lvs_setup

https://gerrit.wikimedia.org/r/1182975

gerritbot added a comment.Sep 18 2025, 9:15 PM

Change #1182977 merged by BCornwall:

[operations/puppet@production] wdqs: (step 4) remove from LBs and wdqs backends

https://gerrit.wikimedia.org/r/1182977

Stashbot added a comment.Sep 18 2025, 9:18 PM

Mentioned in SAL (#wikimedia-operations) [2025-09-18T21:18:34Z] <brett> Restarting pybal on secondary eqiad/codfw lvs servers - T395772

Stashbot added a comment.Sep 18 2025, 9:25 PM

Mentioned in SAL (#wikimedia-operations) [2025-09-18T21:25:37Z] <brett> Restarting pybal on low-traffic eqiad/codfw lvs servers - T395772

Stashbot added a comment.Sep 18 2025, 9:30 PM

Mentioned in SAL (#wikimedia-operations) [2025-09-18T21:30:45Z] <brett> Deleting wdqs, wdqs-heavy-queries, and wdqs-ssl ipvs services from A:lvs-secondary-codfw - T395772

Stashbot added a comment.Sep 18 2025, 9:32 PM

Mentioned in SAL (#wikimedia-operations) [2025-09-18T21:32:07Z] <brett> Deleting wdqs, wdqs-heavy-queries, and wdqs-ssl ipvs services from A:lvs-secondary-eqiad - T395772

Stashbot added a comment.Sep 18 2025, 9:33 PM

Mentioned in SAL (#wikimedia-operations) [2025-09-18T21:33:39Z] <brett> Deleting wdqs, wdqs-heavy-queries, and wdqs-ssl ipvs services from A:lvs-low-traffic-codfw - T395772

Stashbot added a comment.Sep 18 2025, 9:34 PM

Mentioned in SAL (#wikimedia-operations) [2025-09-18T21:34:21Z] <brett> Deleting wdqs, wdqs-heavy-queries, and wdqs-ssl ipvs services from A:lvs-low-traffic-eqiad - T395772

gerritbot added a comment.Sep 18 2025, 9:36 PM

Change #1182978 merged by BCornwall:

[operations/puppet@production] wdqs: (steps 5,6) => final removal

https://gerrit.wikimedia.org/r/1182978

Maintenance_bot removed a project: Patch-For-Review.Sep 18 2025, 10:31 PM
gerritbot added a comment.Sep 18 2025, 10:59 PM

Change #1189595 had a related patch set uploaded (by Ryan Kemper; author: Ryan Kemper):

[operations/puppet@production] wdqs: remove ipip

https://gerrit.wikimedia.org/r/1189595

gerritbot added a project: Patch-For-Review.Sep 18 2025, 10:59 PM
gerritbot added a comment.Sep 18 2025, 11:01 PM

Change #1189595 merged by Ryan Kemper:

[operations/puppet@production] wdqs: remove ipip

https://gerrit.wikimedia.org/r/1189595

gerritbot added a comment.Sep 18 2025, 11:16 PM

Change #1189600 had a related patch set uploaded (by Ryan Kemper; author: Ryan Kemper):

[operations/puppet@production] wdqs: shift old wdqs-public hosts to test

https://gerrit.wikimedia.org/r/1189600

gerritbot added a comment.Sep 18 2025, 11:24 PM

Change #1189600 merged by Ryan Kemper:

[operations/puppet@production] wdqs: shift old wdqs-public hosts to test

https://gerrit.wikimedia.org/r/1189600

ops-monitoring-bot added a comment.Sep 18 2025, 11:26 PM

Cookbook cookbooks.sre.hosts.reimage was started by ryankemper@cumin2002 for host wdqs2016.codfw.wmnet with OS bullseye

Maintenance_bot removed a project: Patch-For-Review.Sep 18 2025, 11:31 PM
ops-monitoring-bot added a comment.Sep 19 2025, 3:44 AM

Cookbook cookbooks.sre.hosts.reimage was started by ryankemper@cumin2002 for host wdqs2017.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Sep 19 2025, 5:17 AM

Cookbook cookbooks.sre.hosts.reimage started by ryankemper@cumin2002 for host wdqs2017.codfw.wmnet with OS bullseye executed with errors:

  • wdqs2017 (FAIL)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Host successfully migrated to the new VLAN
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console wdqs2017.codfw.wmnet" to get a root shell, but depending on the failure this may not work.
Stashbot added a comment.Sep 21 2025, 6:15 PM

Mentioned in SAL (#wikimedia-operations) [2025-09-21T18:15:27Z] <ryankemper@cumin2002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on wdqs[2009,2016].codfw.wmnet,wdqs[1018-1020].eqiad.wmnet with reason: T395772

gerritbot added a comment.Sep 21 2025, 6:26 PM

Change #1189979 had a related patch set uploaded (by Ryan Kemper; author: Ryan Kemper):

[operations/puppet@production] wdqs: point to wdqs-main svc for maxlag

https://gerrit.wikimedia.org/r/1189979

gerritbot added a project: Patch-For-Review.Sep 21 2025, 6:26 PM
gerritbot added a comment.Sep 21 2025, 6:38 PM

Change #1189979 merged by Ryan Kemper:

[operations/puppet@production] wdqs: point to wdqs-main svc for maxlag

https://gerrit.wikimedia.org/r/1189979

Stashbot added a comment.Sep 21 2025, 6:40 PM

Mentioned in SAL (#wikimedia-operations) [2025-09-21T18:40:08Z] <ryankemper> T395772 Merged https://gerrit.wikimedia.org/r/c/operations/puppet/+/1189979 to fix puppet failures on deploy servers

Maintenance_bot removed a project: Patch-For-Review.Sep 21 2025, 7:30 PM
ops-monitoring-bot added a comment.Sep 22 2025, 9:31 PM

Cookbook cookbooks.sre.hosts.reimage was started by ryankemper@cumin2002 for host wdqs2017.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Sep 22 2025, 10:51 PM

Cookbook cookbooks.sre.hosts.reimage started by ryankemper@cumin2002 for host wdqs2017.codfw.wmnet with OS bullseye executed with errors:

  • wdqs2017 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console wdqs2017.codfw.wmnet" to get a root shell, but depending on the failure this may not work.
RKemper added a comment.Sep 23 2025, 6:58 PM
RKemper closed this task as Resolved.Sep 23 2025, 7:24 PM
RKemper moved this task from In Progress to Done on the Data-Platform-SRE (2025.09.05 - 2025.09.26) board.

Removed via sudo -E cumin 'A:config-master' 'rm -fv /srv/config-master/pybal/*/wdqs':

ryankemper@cumin2002:~$ sudo -E cumin 'A:config-master' 'rm -fv /srv/config-master/pybal/*/wdqs'
2 hosts will be targeted:
config-master2001.codfw.wmnet,config-master1001.eqiad.wmnet
OK to proceed on 2 hosts? Enter the number of affected hosts to confirm or "q" to quit: 2
===== NODE GROUP =====
(2) config-master2001.codfw.wmnet,config-master1001.eqiad.wmnet
----- OUTPUT of 'rm -fv /srv/conf...ter/pybal/*/wdqs' -----
removed '/srv/config-master/pybal/codfw/wdqs'
removed '/srv/config-master/pybal/eqiad/wdqs'
================
PASS |███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 100% (2/2) [00:00<00:00,  2.83hosts/s]
FAIL |                                                                                                                                               |   0% (0/2) [00:00<?, ?hosts/s]
100.0% (2/2) success ratio (>= 100.0% threshold) for command: 'rm -fv /srv/conf...ter/pybal/*/wdqs'.
100.0% (2/2) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
Stashbot added a comment.Sep 25 2025, 9:27 PM

Mentioned in SAL (#wikimedia-operations) [2025-09-25T21:27:51Z] <ryankemper@cumin2002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5 days, 0:00:00 on wdqs[2009,2016].codfw.wmnet,wdqs[1018-1020].eqiad.wmnet with reason: T395772

gerritbot added a comment.Sep 25 2025, 9:32 PM

Change #1191513 had a related patch set uploaded (by Ryan Kemper; author: Ryan Kemper):

[operations/puppet@production] wdqs: these hosts no longer in wdqs-public

https://gerrit.wikimedia.org/r/1191513

gerritbot added a project: Patch-For-Review.Sep 25 2025, 9:32 PM
gerritbot added a comment.Sep 25 2025, 9:42 PM

Change #1191513 merged by Ryan Kemper:

[operations/puppet@production] wdqs: these hosts no longer in wdqs-public

https://gerrit.wikimedia.org/r/1191513

Stashbot added a comment.Sep 25 2025, 9:43 PM

Mentioned in SAL (#wikimedia-operations) [2025-09-25T21:43:16Z] <ryankemper@cumin2002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5 days, 0:00:00 on wdqs[2009,2016].codfw.wmnet,wdqs[1018-1020].eqiad.wmnet with reason: T395772

gerritbot added a comment.Sep 25 2025, 10:16 PM

Change #1191525 had a related patch set uploaded (by Ryan Kemper; author: Ryan Kemper):

[operations/puppet@production] wdqs: shift old full graph hosts to wdqs-main

https://gerrit.wikimedia.org/r/1191525

Gehel moved this task from Done to Reported on the Data-Platform-SRE (2025.09.05 - 2025.09.26) board.Sep 26 2025, 9:35 AM
gerritbot added a comment.Sep 29 2025, 9:17 PM

Change #1191525 merged by Ryan Kemper:

[operations/puppet@production] wdqs: shift old full graph hosts to new roles

https://gerrit.wikimedia.org/r/1191525

Maintenance_bot removed a project: Patch-For-Review.Sep 29 2025, 9:32 PM
bking changed the status of subtask T405978: Re-image remaining full graph hosts to post-graph-split roles from Open to In Progress.Sep 30 2025, 8:46 PM
Gehel closed subtask T405978: Re-image remaining full graph hosts to post-graph-split roles as Resolved.Oct 10 2025, 8:33 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits