Page MenuHomePhabricator
Create Task
Maniphest T415073

Cleanup after decommission of the WDQS full graph endpoint
Open, In Progress, HighPublic
Actions

Description

At least the following needs to be cleanup:

  • puppet role / profiles / configuration related to the full graph
  • DNS entries
  • UI minisite

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
cumin: repurpose wdqs-public, add wdqs-internaloperations/puppetproduction+3 -2
wdqs: nuke dead config from legacy-full decomoperations/puppetproduction+0 -13
wdqs: drop dangling query-legacy-full helm refsoperations/deployment-chartsmaster+0 -3
wdqs: allow NFS mount from wdqs2009operations/puppetproduction+1 -0
wdqs: repurpose wdqs2009 to test blazegraph alternativesoperations/puppetproduction+4 -5
wdqs: remove stale legacy-full-gui release entryoperations/deployment-chartsmaster+0 -2
wdqs: remove query-legacy-fulloperations/deployment-chartsmaster+0 -44
wdqs: cleanup code related to query-legacy-full.wikidata.orgoperations/puppetproduction+2 -122
wdqs: remove query-legacy-full.wikidata.org - end of lifeoperations/dnsmaster+0 -1
Customize query in gerrit

Related Objects
Search...

Event Timeline

Gehel created this task.Jan 20 2026, 2:28 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 20 2026, 2:28 PM
Maintenance_bot added a project: Wikidata.Jan 20 2026, 2:30 PM
gmodena subscribed.Jan 20 2026, 2:39 PM
BTracy-WMF subscribed.Jan 20 2026, 3:04 PM
Gehel triaged this task as High priority.Jan 20 2026, 3:27 PM
Gehel moved this task from Backlog - project to Backlog - operations on the Data-Platform-SRE (2026.01.05 - 2026.01.23) board.
BTracy-WMF moved this task from Incoming to Operations/SRE on the Wikidata-Query-Service board.Jan 22 2026, 2:12 PM
Gehel edited projects, added Data-Platform-SRE (2026.01.23 - 2026.02.13); removed Data-Platform-SRE (2026.01.05 - 2026.01.23).Jan 23 2026, 10:04 AM
Gehel moved this task from Backlog - project to Backlog - operations on the Data-Platform-SRE (2026.01.23 - 2026.02.13) board.
Gehel removed Gehel as the assignee of this task.Feb 3 2026, 3:39 PM
Gehel edited projects, added Data-Platform-SRE (2026-02-13 - 2026-03-06); removed Data-Platform-SRE (2026.01.23 - 2026.02.13).Feb 16 2026, 3:36 PM
Gehel moved this task from Backlog - project to Backlog - operations on the Data-Platform-SRE (2026-02-13 - 2026-03-06) board.
BTracy-WMF added a comment.EditedFeb 25 2026, 4:10 PM

Grafana is showing data for performance metrics for the legacy endpoint and it appears queries still run on query-legacy-full.wikidata.org (example). Is this because the cleanup work is still in progress?

Gehel added a comment.Mar 4 2026, 9:01 AM
In T415073#11651358, @BTracy-WMF wrote:

Grafana is showing data for performance metrics for the legacy endpoint and it appears queries still run on query-legacy-full.wikidata.org (example). Is this because the cleanup work is still in progress?

Damn, Blazegraph was restarted on that node. Let me kill it again.

Stashbot mentioned this in T411410: Decommission WDQS full graph endpoint (wdqs2009).Mar 4 2026, 9:03 AM

Mentioned in SAL (#wikimedia-operations) [2026-03-04T09:03:24Z] <gehel> switching off Blazegraph on wdqs2009 (legacy full graph endpoint is end of life) - T411410 / T415073

gerritbot added a comment.Mar 4 2026, 9:05 AM

Change #1247926 had a related patch set uploaded (by Gehel; author: Gehel):

[operations/dns@master] wdqs: remove query-legacy-full.wikidata.org - end of life

https://gerrit.wikimedia.org/r/1247926

gerritbot added a project: Patch-For-Review.Mar 4 2026, 9:05 AM
gerritbot added a comment.Mar 4 2026, 9:17 AM

Change #1247933 had a related patch set uploaded (by Gehel; author: Gehel):

[operations/puppet@production] wdqs: cleanup code related to query-legacy-full.wikidata.org

https://gerrit.wikimedia.org/r/1247933

Gehel moved this task from Backlog - operations to In Progress on the Data-Platform-SRE (2026-02-13 - 2026-03-06) board.Mar 4 2026, 10:04 AM
gerritbot added a comment.Mar 4 2026, 10:04 AM

Change #1247947 had a related patch set uploaded (by Gehel; author: Gehel):

[operations/deployment-charts@master] wdqs: remove query-legay-full

https://gerrit.wikimedia.org/r/1247947

Gehel moved this task from In Progress to Needs Review on the Data-Platform-SRE (2026-02-13 - 2026-03-06) board.Mar 4 2026, 10:09 AM
gerritbot added a comment.Mar 5 2026, 8:27 AM

Change #1247926 merged by Gehel:

[operations/dns@master] wdqs: remove query-legacy-full.wikidata.org - end of life

https://gerrit.wikimedia.org/r/1247926

Gehel added a comment.Mar 5 2026, 8:32 AM

I've removed the DNS records, so query-legacy-full.wd.o is truely unreachable now.

gerritbot added a comment.Mar 6 2026, 5:43 AM

Change #1247933 merged by Ryan Kemper:

[operations/puppet@production] wdqs: cleanup code related to query-legacy-full.wikidata.org

https://gerrit.wikimedia.org/r/1247933

gerritbot added a comment.Mar 6 2026, 5:58 AM

Change #1247947 merged by jenkins-bot:

[operations/deployment-charts@master] wdqs: remove query-legacy-full

https://gerrit.wikimedia.org/r/1247947

gerritbot added a comment.Mar 6 2026, 6:26 AM

Change #1248694 had a related patch set uploaded (by Ryan Kemper; author: Ryan Kemper):

[operations/deployment-charts@master] wdqs: remove stale legacy-full-gui release entry

https://gerrit.wikimedia.org/r/1248694

RKemper subscribed.Mar 6 2026, 6:39 AM

Merged puppet cleanup (https://gerrit.wikimedia.org/r/c/operations/puppet/+/1247933)
and deployment-charts cleanup (https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1247947).
Also ran helmfile apply across staging/eqiad/codfw to teardown the
wikidata-query-legacy-full-gui helm release.

Follow-up patch to remove stale releases block entry:
https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1248694

AC Checklist status:

  • puppet role / profiles / configuration related to the full graph
  • DNS entries (already removed — query-legacy-full.wikidata.org no longer resolves)
  • UI minisite (helmfile reference removed + helm release uninstalled)
RKemper updated the task description. (Show Details)Mar 6 2026, 6:39 AM
Gehel edited projects, added Data-Platform-SRE (2026-03-06 - 2026-03-27); removed Data-Platform-SRE (2026-02-13 - 2026-03-06).Mar 6 2026, 2:01 PM
Gehel moved this task from Backlog - project to Needs Review on the Data-Platform-SRE (2026-03-06 - 2026-03-27) board.
gerritbot added a comment.Mar 6 2026, 10:04 PM

Change #1248694 merged by jenkins-bot:

[operations/deployment-charts@master] wdqs: remove stale legacy-full-gui release entry

https://gerrit.wikimedia.org/r/1248694

Maintenance_bot removed a project: Patch-For-Review.Mar 6 2026, 10:31 PM
ops-monitoring-bot added a comment.Mar 6 2026, 10:39 PM

Cookbook cookbooks.sre.hosts.reimage was started by ryankemper@cumin2002 for host wdqs2009.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Mar 6 2026, 11:29 PM

Cookbook cookbooks.sre.hosts.reimage started by ryankemper@cumin2002 for host wdqs2009.codfw.wmnet with OS bullseye completed:

  • wdqs2009 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Host successfully migrated to the new VLAN
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202603062313_ryankemper_95432_wdqs2009.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
gerritbot added a comment.Mar 9 2026, 1:42 PM

Change #1249298 had a related patch set uploaded (by Bking; author: Bking):

[operations/puppet@production] wdqs: repurpose wdqs2009 to test blazegraph alternatives

https://gerrit.wikimedia.org/r/1249298

gerritbot added a project: Patch-For-Review.Mar 9 2026, 1:42 PM

Change #1249298 merged by Bking:

[operations/puppet@production] wdqs: repurpose wdqs2009 to test blazegraph alternatives

https://gerrit.wikimedia.org/r/1249298

ops-monitoring-bot added a comment.Mar 9 2026, 1:54 PM

Cookbook cookbooks.sre.hosts.reimage was started by bking@cumin2002 for host wdqs2009.codfw.wmnet with OS bullseye

Maintenance_bot removed a project: Patch-For-Review.Mar 9 2026, 2:32 PM
ops-monitoring-bot added a comment.Mar 9 2026, 2:49 PM

Cookbook cookbooks.sre.hosts.reimage started by bking@cumin2002 for host wdqs2009.codfw.wmnet with OS bullseye executed with errors:

  • wdqs2009 (FAIL)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202603091420_bking_860931_wdqs2009.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console wdqs2009.codfw.wmnet" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.Mar 9 2026, 2:50 PM

Cookbook cookbooks.sre.hosts.reimage was started by bking@cumin2002 for host wdqs2009.codfw.wmnet with OS bookworm

ops-monitoring-bot added a comment.Mar 10 2026, 6:14 AM

Cookbook cookbooks.sre.hosts.reimage started by bking@cumin2002 for host wdqs2009.codfw.wmnet with OS bookworm executed with errors:

  • wdqs2009 (FAIL)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202603091512_bking_874523_wdqs2009.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console wdqs2009.codfw.wmnet" to get a root shell, but depending on the failure this may not work.
gerritbot added a comment.Mar 11 2026, 8:55 PM

Change #1250683 had a related patch set uploaded (by Bking; author: Bking):

[operations/puppet@production] wdqs: allow NFS mount from wdqs2009

https://gerrit.wikimedia.org/r/1250683

gerritbot added a project: Patch-For-Review.Mar 11 2026, 8:55 PM
gerritbot added a comment.Mar 12 2026, 1:14 PM

Change #1250683 merged by Bking:

[operations/puppet@production] wdqs: allow NFS mount from wdqs2009

https://gerrit.wikimedia.org/r/1250683

Maintenance_bot removed a project: Patch-For-Review.Mar 12 2026, 1:32 PM
Gehel edited projects, added Data-Platform-SRE (2026-03-27 - 2026-04-17); removed Data-Platform-SRE (2026-03-06 - 2026-03-27).Mar 27 2026, 9:52 AM
Gehel moved this task from Backlog - project to Needs Review on the Data-Platform-SRE (2026-03-27 - 2026-04-17) board.
Gehel edited projects, added Data-Platform-SRE (2026-04-24 - 2026-05-15); removed Data-Platform-SRE (2026-03-27 - 2026-04-17).Fri, Apr 24, 9:16 AM
Gehel moved this task from Backlog - project to Needs Review on the Data-Platform-SRE (2026-04-24 - 2026-05-15) board.
RKemper closed this task as Resolved.Tue, Apr 28, 3:39 PM
RKemper claimed this task.
RKemper moved this task from Needs Review to Done on the Data-Platform-SRE (2026-04-24 - 2026-05-15) board.
RKemper reopened this task as In Progress.Tue, Apr 28, 3:51 PM
RKemper moved this task from Done to In Progress on the Data-Platform-SRE (2026-04-24 - 2026-05-15) board.

I spoke too soon. Came across some entries in deployment-charts, as well as some dangling puppet aliases. Working on a more rigorous enumeration of remaining AC

gerritbot added a comment.Tue, Apr 28, 8:08 PM

Change #1278562 had a related patch set uploaded (by Ryan Kemper; author: Ryan Kemper):

[operations/deployment-charts@master] wdqs: drop dangling query-legacy-full helm refs

https://gerrit.wikimedia.org/r/1278562

gerritbot added a project: Patch-For-Review.Tue, Apr 28, 8:08 PM
gerritbot added a comment.Tue, Apr 28, 9:15 PM

Change #1278602 had a related patch set uploaded (by Ryan Kemper; author: Ryan Kemper):

[operations/puppet@production] wdqs: nuke dead config from legacy-full decom

https://gerrit.wikimedia.org/r/1278602

gerritbot added a comment.Tue, Apr 28, 9:15 PM

Change #1278603 had a related patch set uploaded (by Ryan Kemper; author: Ryan Kemper):

[operations/puppet@production] cumin: repurpose wdqs-public, add wdqs-internal

https://gerrit.wikimedia.org/r/1278603

gerritbot added a comment.Tue, Apr 28, 9:48 PM

Change #1278602 merged by Bking:

[operations/puppet@production] wdqs: nuke dead config from legacy-full decom

https://gerrit.wikimedia.org/r/1278602

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits