Page MenuHomePhabricator
Create Task
Maniphest T432148

Tear down the datahubsearch service and its constituent VMs
Closed, ResolvedPublic
Actions

Description

As part of T402408: Upgrade Datahub to a more recent version we migrated the OpenSearch cluster behind datahub.wikimedia.org to Kubernetes, running on the dse-k8s-eqiad cluster.

Because of this, we no longer need the datahubsearch service that is provided by the three Gantei virtual machines: datahubsearch100[1-3].eqiad.wmnet

This ticket will track the decommissioning of the service and the VMs.

We need to remove datahubsearch from at least one OpenSearch dashboard as well.

Details

Related Changes in Gerrit:
SubjectAuthorRepoBranchLines +/-
datahubsearch: remove references to recently-decommed clusterBkingoperations/puppetproduction+0 -4
datahubsearch: move to 'insetup' roleBkingoperations/puppetproduction+1 -1
datahubsearch: remove references to Puppet plansBkingoperations/puppetproduction+0 -113
datahub search: move service to 'service_setup' state.Bkingoperations/puppetproduction+1 -8
datahubsearch: move service from production to lvs_setupBkingoperations/puppetproduction+1 -1
datahubsearch: remove DNS recordBkingoperations/dnsmaster+0 -1
opensearch: remove references to soon-to-be-decom'd serviceBkingoperations/alertsmaster+5 -6
Customize query in gerrit

Related Objects

Event Timeline

BTullis created this task.Tue, Jul 14, 4:35 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptTue, Jul 14, 4:35 PM
bking subscribed.Wed, Jul 15, 9:58 PM
ops-monitoring-bot added a comment.Wed, Jul 15, 10:01 PM

Icinga downtime and Alertmanager silence (ID=a24bd063-e76d-49af-86b5-aec247d1d3d6) set by bking@cumin2003 for 4:00:00 on 3 host(s) and their services with reason: Using datahubsearch1001 to test bookworm reimages

datahubsearch[1001-1003].eqiad.wmnet
bking mentioned this in T430879: Migrate WCQS to Bookworm or later.Wed, Jul 15, 10:05 PM
ops-monitoring-bot added a comment.Wed, Jul 15, 10:16 PM

Icinga downtime and Alertmanager silence (ID=21dd99da-77d1-4c0e-9bf2-e6fc5c0efd56) set by bking@cumin2003 for 7 days, 0:00:00 on 2 host(s) and their services with reason: Using datahubsearch1001 to test bookworm reimages

datahubsearch[1002-1003].eqiad.wmnet
bking changed the task status from Open to In Progress.Wed, Jul 15, 10:36 PM
bking claimed this task.
bking triaged this task as Low priority.
bking edited projects, added: Data-Platform-SRE (2026-07-03 - 2026-07-31); removed: Data-Platform-SRE.
bking moved this task from Backlog - project to In Progress on the Data-Platform-SRE (2026-07-03 - 2026-07-31) board.
gerritbot added a comment.Thu, Jul 16, 7:36 PM

Change #1311530 had a related patch set uploaded (by Bking; author: Bking):

[operations/puppet@production] datahubsearch: move to 'insetup' role

https://gerrit.wikimedia.org/r/1311530

gerritbot added a project: Patch-For-Review.Thu, Jul 16, 7:36 PM
gerritbot added a comment.Thu, Jul 16, 8:03 PM

Change #1311533 had a related patch set uploaded (by Bking; author: Bking):

[operations/puppet@production] datahubsearch: remove references to Puppet plans

https://gerrit.wikimedia.org/r/1311533

gerritbot added a comment.Thu, Jul 16, 8:43 PM

Change #1311538 had a related patch set uploaded (by Bking; author: Bking):

[operations/dns@master] datahubsearch: remove DNS record

https://gerrit.wikimedia.org/r/1311538

gerritbot added a comment.Thu, Jul 16, 9:00 PM

Change #1311542 had a related patch set uploaded (by Bking; author: Bking):

[operations/alerts@master] opensearch: remove references to soon-to-be-decom'd service

https://gerrit.wikimedia.org/r/1311542

gerritbot added a comment.Fri, Jul 17, 6:25 PM

Change #1311542 merged by Bking:

[operations/alerts@master] opensearch: remove references to soon-to-be-decom'd service

https://gerrit.wikimedia.org/r/1311542

gerritbot added a comment.Fri, Jul 17, 6:28 PM

Change #1311538 merged by Bking:

[operations/dns@master] datahubsearch: remove DNS record

https://gerrit.wikimedia.org/r/1311538

gerritbot added a comment.Fri, Jul 17, 6:50 PM

Change #1311899 had a related patch set uploaded (by Bking; author: Bking):

[operations/puppet@production] datahubsearch: move service from production to lvs_setup

https://gerrit.wikimedia.org/r/1311899

gerritbot added a comment.Mon, Jul 20, 2:28 PM

Change #1312517 had a related patch set uploaded (by Bking; author: Bking):

[operations/puppet@production] datahub search: move service to 'service_setup' state.

https://gerrit.wikimedia.org/r/1312517

gerritbot added a comment.Mon, Jul 20, 2:38 PM

Change #1311899 merged by Bking:

[operations/puppet@production] datahubsearch: move service from production to lvs_setup

https://gerrit.wikimedia.org/r/1311899

bking updated the task description. (Show Details)Mon, Jul 20, 6:19 PM
gerritbot added a comment.Tue, Jul 21, 1:40 PM

Change #1312517 merged by Bking:

[operations/puppet@production] datahub search: move service to 'service_setup' state.

https://gerrit.wikimedia.org/r/1312517

gerritbot added a comment.Tue, Jul 21, 3:00 PM

Change #1311533 merged by Bking:

[operations/puppet@production] datahubsearch: remove references to Puppet plans

https://gerrit.wikimedia.org/r/1311533

gerritbot added a comment.Tue, Jul 21, 10:05 PM

Change #1311530 merged by Bking:

[operations/puppet@production] datahubsearch: move to 'insetup' role

https://gerrit.wikimedia.org/r/1311530

Maintenance_bot removed a project: Patch-For-Review.Tue, Jul 21, 10:31 PM
ops-monitoring-bot added a comment.Wed, Jul 22, 1:53 PM

cookbooks.sre.hosts.decommission executed by bking@cumin2003 for hosts: datahubsearch[1002-1003].eqiad.wmnet

  • datahubsearch1002.eqiad.wmnet (PASS)
    • Downtimed host on Icinga/Alertmanager
    • Found Ganeti VM
    • VM shutdown
    • Started forced sync of VMs in Ganeti cluster eqiad to Netbox
    • Removed from DebMonitor
    • Removed from Puppet server and PuppetDB
    • VM removed
    • Started forced sync of VMs in Ganeti cluster eqiad to Netbox
  • datahubsearch1003.eqiad.wmnet (PASS)
    • Downtimed host on Icinga/Alertmanager
    • Found Ganeti VM
    • VM shutdown
    • Started forced sync of VMs in Ganeti cluster eqiad to Netbox
    • Removed from DebMonitor
    • Removed from Puppet server and PuppetDB
    • VM removed
    • Started forced sync of VMs in Ganeti cluster eqiad to Netbox
bking added a comment.Wed, Jul 22, 2:12 PM

datahubsearch1001 (which had fallen out of the PuppetDB ) is still in process of decommissioning:

[bking@ganeti1048] ~$ sudo gnt-instance list | grep datahubsearch
datahubsearch1001.eqiad.wmnet       kvm        debootstrap+default ganeti1057.eqiad.wmnet ADMIN_down      -

Assuming the decom cookbook finishes cleanly, this should be the last step in getting rid of the cluster.

ops-monitoring-bot added a comment.Wed, Jul 22, 2:32 PM

cookbooks.sre.hosts.decommission executed by bking@cumin2003 for hosts: datahubsearch1001.eqiad.wmnet

  • datahubsearch1001.eqiad.wmnet (WARN)
    • Host not found on Icinga, unable to downtime it
    • Found Ganeti VM
    • VM shutdown
    • Started forced sync of VMs in Ganeti cluster eqiad to Netbox
    • Removed from DebMonitor
    • Removed from Puppet server and PuppetDB
    • VM removed
    • Started forced sync of VMs in Ganeti cluster eqiad to Netbox
bking closed this task as Resolved.EditedWed, Jul 22, 2:58 PM
bking moved this task from In Progress to Done on the Data-Platform-SRE (2026-07-03 - 2026-07-31) board.

Per the above message, all VMs and their associated configs have been removed. As such, I'm closing this ticket. However, do feel free to reopen if we missed anything.

I still need to remove the hostnames from Puppet's site.pp.

bking reopened this task as In Progress.Wed, Jul 22, 2:59 PM
bking moved this task from Done to In Progress on the Data-Platform-SRE (2026-07-03 - 2026-07-31) board.
gerritbot added a comment.Wed, Jul 22, 3:02 PM

Change #1313978 had a related patch set uploaded (by Bking; author: Bking):

[operations/puppet@production] datahubsearch: remove references to recently-decommed cluster

https://gerrit.wikimedia.org/r/1313978

gerritbot added a project: Patch-For-Review.Wed, Jul 22, 3:02 PM
gerritbot added a comment.Wed, Jul 22, 3:11 PM

Change #1313978 merged by Bking:

[operations/puppet@production] datahubsearch: remove references to recently-decommed cluster

https://gerrit.wikimedia.org/r/1313978

bking closed this task as Resolved.Wed, Jul 22, 3:31 PM
bking moved this task from In Progress to Done on the Data-Platform-SRE (2026-07-03 - 2026-07-31) board.

OK, I believe that was the final step. Closing...

Maintenance_bot removed a project: Patch-For-Review.Wed, Jul 22, 3:31 PM
Gehel moved this task from Done to Reported on the Data-Platform-SRE (2026-07-03 - 2026-07-31) board.Fri, Aug 7, 7:53 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits