Page MenuHomePhabricator
Create Task
Maniphest T347278

Decommission ORES configurations and servers
Closed, ResolvedPublic3 Estimated Story Points
Actions

Description

Now that ores-legacy handles all the traffic for ores.wikimedia.org, we can start decomming it.

The starting point is https://wikitech.wikimedia.org/wiki/Server_Lifecycle#Remove_from_production

High level idea:

  • Remove old icinga alerts etc.. Set LVS checks to not page if all nodes are down.
  • Remove ores nodes from pybal, plus its config (VIP, etc..). This requires coordination with Traffic since some pybal restarts are probably needed (to update LVS etc..).
  • Shutdown all ores nodes + downtime.
  • Clean up DNS records.
  • Clean up puppet from ores classes and configs (including POSIX groups etc..) (the ores-admin group was only ever applied to hosts which now get fully decommissioned, as such the group entry can be removed entirely and the GID added to the reclaim list in L71 of data.yaml)
  • Cleanup TLS certificates for ores.discovery.wmnet
  • Check in Horizon for old VMs etc.. We should probably shutdown everything, and close projects that we don't need (so we release capacity etc..).
  • Update the Wikitech/Mediawiki documentation
  • Delete old grafana dashboards
  • Archive ores-related repositories - tracked in T349632

Details

SubjectRepoBranchLines +/-
profiles: Remove more ORES leftoversoperations/puppetproduction+0 -1
hiera: clean up more ORES leftoverslabs/privatemaster+0 -3
Clean up additional ORES leftoversoperations/puppetproduction+0 -16
Clean up ores configs not used anymoreoperations/puppetproduction+0 -11
cloud: Remove ores-beta ATS settingsoperations/puppetproduction+0 -6
contactgroups: remove old team-scoringoperations/puppetproduction+0 -5
admin: remove ores-admins groupoperations/puppetproduction+1 -10
profile::prometheus::ops: remove ORES Redis configsoperations/puppetproduction+0 -13
Remove ORES configs and clustersoperations/puppetproduction+0 -12
Remove ORES deployment settingsoperations/puppetproduction+0 -14
profile::logstash: remove ORES configsoperations/puppetproduction+0 -58
profile::httpbb: remove ores_test configsoperations/puppetproduction+0 -3
Remove ORES roles and configsoperations/puppetproduction+0 -1 K
Remove ores listener from mediawiki fixturesoperations/deployment-chartsmaster+0 -2
role::redis::misc::{master,slave}: remove ORES configsoperations/puppetproduction+2 -8
icinga/nagios: remove check_ores*operations/puppetproduction+0 -48
hiera/services: add service for recommendation-api-ngoperations/puppetproduction+31 -0
conftool-data: Add entry for recommendation-api-ngoperations/puppetproduction+1 -0
Remove ores.svc.{eqiad,codfw}.wmnet recordsoperations/dnsmaster+0 -2
ORES: remove ORES from Envoy listeners listoperations/puppetproduction+0 -5
Services: Remove pybal/LVS entry for ORESoperations/puppetproduction+0 -57
services/lvs: Turn down ORES LVS setupoperations/puppetproduction+1 -7
services: Move ORES to state lvs_setup for turndownoperations/puppetproduction+1 -1
wmnet: Remove ORES discovery entryoperations/dnsmaster+0 -2
wmnet: Remove ORES discovery entryoperations/dnsmaster+0 -4
Avoid pages for ores.discovery.wmnetoperations/puppetproduction+1 -0
Show related patches Customize query in gerrit

Related Objects

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
gerritbot added a comment.Sep 28 2023, 1:35 PM

Change 961802 merged by Klausman:

[operations/dns@master] wmnet: Remove ORES discovery entry

https://gerrit.wikimedia.org/r/961802

gerritbot added a comment.Sep 28 2023, 1:39 PM

Change 961799 merged by Klausman:

[operations/puppet@production] services: Move ORES to state lvs_setup for turndown

https://gerrit.wikimedia.org/r/961799

gerritbot added a comment.Sep 28 2023, 1:48 PM

Change 961805 had a related patch set uploaded (by Klausman; author: Klausman):

[operations/puppet@production] services/lvs: Turn down ORES LVS setup

https://gerrit.wikimedia.org/r/961805

gerritbot added a comment.Sep 28 2023, 1:53 PM

Change 961805 merged by Klausman:

[operations/puppet@production] services/lvs: Turn down ORES LVS setup

https://gerrit.wikimedia.org/r/961805

Stashbot added a comment.Sep 28 2023, 2:00 PM

Mentioned in SAL (#wikimedia-operations) [2023-09-28T14:00:51Z] <klausman> restarted pybal on lvs1020 and lvs2014 (LVS low-traffic backups) for T347278 (ORES turndown)

Stashbot added a comment.Sep 28 2023, 2:13 PM

Mentioned in SAL (#wikimedia-operations) [2023-09-28T14:13:21Z] <klausman> restarting pybal on lvs1019 and lvs2013 (LVS low-traffic actives) for T347278 (ORES turndown)

gerritbot added a comment.Sep 28 2023, 2:34 PM

Change 961791 merged by Klausman:

[operations/puppet@production] Services: Remove pybal/LVS entry for ORES

https://gerrit.wikimedia.org/r/961791

gerritbot added a comment.Sep 28 2023, 2:42 PM

Change 961825 had a related patch set uploaded (by Klausman; author: Klausman):

[operations/puppet@production] ORES: remove profile::services_proxy::envoy::enabled_listeners role

https://gerrit.wikimedia.org/r/961825

gerritbot added a comment.Sep 28 2023, 2:56 PM

Change 961825 merged by Klausman:

[operations/puppet@production] ORES: remove ORES from Envoy listeners list

https://gerrit.wikimedia.org/r/961825

klausman updated the task description. (Show Details)Sep 28 2023, 3:03 PM
Maintenance_bot removed a project: Patch-For-Review.Sep 28 2023, 3:10 PM
MoritzMuehlenhoff updated the task description. (Show Details)Sep 28 2023, 3:13 PM
MoritzMuehlenhoff updated the task description. (Show Details)
gerritbot added a comment.Oct 2 2023, 3:42 PM

Change 962642 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/dns@master] Remove ores.svc.{eqiad,codfw}.wmnet records

https://gerrit.wikimedia.org/r/962642

gerritbot added a project: Patch-For-Review.Oct 2 2023, 3:43 PM
gerritbot added a comment.Oct 3 2023, 8:41 AM

Change 962642 merged by Elukey:

[operations/dns@master] Remove ores.svc.{eqiad,codfw}.wmnet records

https://gerrit.wikimedia.org/r/962642

Maintenance_bot removed a project: Patch-For-Review.Oct 3 2023, 9:11 AM
gerritbot added a comment.Oct 3 2023, 9:18 AM

Change 963009 had a related patch set uploaded (by Klausman; author: Klausman):

[operations/puppet@production] conftool-data: Add entry for recommendation-api-ng

https://gerrit.wikimedia.org/r/963009

gerritbot added a project: Patch-For-Review.Oct 3 2023, 9:18 AM
gerritbot added a comment.Oct 3 2023, 9:24 AM

Change 963009 merged by Klausman:

[operations/puppet@production] conftool-data: Add entry for recommendation-api-ng

https://gerrit.wikimedia.org/r/963009

Maintenance_bot removed a project: Patch-For-Review.Oct 3 2023, 9:30 AM
gerritbot added a comment.Oct 3 2023, 9:43 AM

Change 963013 had a related patch set uploaded (by Klausman; author: Klausman):

[operations/puppet@production] hiera/services: add service for recommendation-api-ng

https://gerrit.wikimedia.org/r/963013

gerritbot added a project: Patch-For-Review.Oct 3 2023, 9:43 AM
gerritbot added a comment.Oct 3 2023, 3:07 PM

Change 963013 abandoned by Klausman:

[operations/puppet@production] hiera/services: add service for recommendation-api-ng

Reason:

Not needed since this is a ml-k8s service

https://gerrit.wikimedia.org/r/963013

Maintenance_bot removed a project: Patch-For-Review.Oct 3 2023, 3:10 PM
ops-monitoring-bot added a comment.Oct 4 2023, 12:46 PM

Icinga downtime and Alertmanager silence (ID=79579710-b671-4886-a85b-afefbf9b3afb) set by klausman@cumin1001 for 90 days, 0:00:00 on 22 host(s) and their services with reason: Downtime for graceful shutdown and later decom

ores[2001-2009].codfw.wmnet,ores[1001-1009].eqiad.wmnet,orespoolcounter[2003-2004].codfw.wmnet,orespoolcounter[1003-1004].eqiad.wmnet
klausman added a comment.Oct 4 2023, 1:03 PM

The machines ores[2001-2009].codfw.wmnet,ores[1001-1009].eqiad.wmnet have been shut down (1001 and 2001 are still running in case we need files from them).

klausman added a comment.Oct 4 2023, 1:23 PM

After discussion on IRC, I have also shutdown 1001 and 2001.

klausman added a comment.Oct 4 2023, 1:41 PM

File T348144 for decomming.

elukey updated the task description. (Show Details)Oct 9 2023, 8:04 AM
elukey updated the task description. (Show Details)Oct 9 2023, 12:55 PM
Volans subscribed.Oct 11 2023, 11:21 AM

@klausman the DNS step is marked as done, but I see the ORES SVC records still existing in Netbox ( https://netbox.wikimedia.org/ipam/ip-addresses/?q=ores ) is that a leftover or pending some other step? (when removed a run of the sre.dns.netbox cookbook is needed)

elukey added a comment.Oct 11 2023, 12:14 PM
In T347278#9242271, @Volans wrote:

@klausman the DNS step is marked as done, but I see the ORES SVC records still existing in Netbox ( https://netbox.wikimedia.org/ipam/ip-addresses/?q=ores ) is that a leftover or pending some other step? (when removed a run of the sre.dns.netbox cookbook is needed)

My bad! Removed the records and ran the cookbook :)

Volans mentioned this in T270071: SVC DNS zonefiles and source of truth.Oct 11 2023, 12:21 PM
gerritbot added a comment.Oct 11 2023, 12:27 PM

Change 965124 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] role::redis::misc::{master,slave}: remove ORES configs

https://gerrit.wikimedia.org/r/965124

gerritbot added a project: Patch-For-Review.Oct 11 2023, 12:27 PM
Stashbot added a comment.Oct 12 2023, 10:38 AM

Mentioned in SAL (#wikimedia-releng) [2023-10-12T10:38:46Z] <elukey> delete ores proxy and instance in deployment-prep - T347278

elukey updated the task description. (Show Details)Oct 12 2023, 10:43 AM
elukey updated the task description. (Show Details)Oct 12 2023, 10:45 AM
gerritbot added a comment.Oct 18 2023, 10:03 AM

Change 965124 merged by Effie Mouzeli:

[operations/puppet@production] role::redis::misc::{master,slave}: remove ORES configs

https://gerrit.wikimedia.org/r/965124

Maintenance_bot removed a project: Patch-For-Review.Oct 18 2023, 10:11 AM
jijiki updated the task description. (Show Details)Oct 18 2023, 10:46 AM
jijiki subscribed.

Data has been flushed from both rdb1011 and rdb2009

gerritbot added a comment.Oct 20 2023, 1:13 PM

Change 967447 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/deployment-charts@master] Remove ores listener from mediawiki fixtures

https://gerrit.wikimedia.org/r/967447

gerritbot added a project: Patch-For-Review.Oct 20 2023, 1:13 PM
gerritbot added a comment.Oct 20 2023, 1:34 PM

Change 967447 merged by jenkins-bot:

[operations/deployment-charts@master] Remove ores listener from mediawiki fixtures

https://gerrit.wikimedia.org/r/967447

Maintenance_bot removed a project: Patch-For-Review.Oct 20 2023, 2:10 PM
elukey added a comment.Oct 23 2023, 9:08 AM
elukey@puppetmaster1001:~$ sudo puppet cert clean ores.discovery.wmnet
Warning: `puppet cert` is deprecated and will be removed in a future release.
   (location: /usr/lib/ruby/vendor_ruby/puppet/application.rb:370:in `run')
Notice: Revoked certificate with serial 6792
Notice: Removing file Puppet::SSL::Certificate ores.discovery.wmnet at '/var/lib/puppet/server/ssl/ca/signed/ores.discovery.wmnet.pem'
Notice: Removing file Puppet::SSL::Certificate ores.discovery.wmnet at '/var/lib/puppet/server/ssl/certs/ores.discovery.wmnet.pem'
elukey updated the task description. (Show Details)Oct 23 2023, 9:11 AM
calbon set the point value for this task to 3.Nov 2 2023, 7:13 PM
calbon moved this task from In Progress to Ready To Go on the Machine-Learning-Team board.
achou triaged this task as Medium priority.Nov 2 2023, 7:28 PM
gerritbot added a comment.Nov 17 2023, 9:55 AM

Change 975213 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] Remove ORES roles and configs

https://gerrit.wikimedia.org/r/975213

gerritbot added a project: Patch-For-Review.Nov 17 2023, 9:55 AM

Change 975214 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] profile::logstash: remove ORES configs

https://gerrit.wikimedia.org/r/975214

gerritbot added a comment.Nov 17 2023, 9:55 AM

Change 975215 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] Remove ORES deployment settings

https://gerrit.wikimedia.org/r/975215

gerritbot added a comment.Nov 17 2023, 9:55 AM

Change 975216 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] Remove ORES configs and clusters

https://gerrit.wikimedia.org/r/975216

gerritbot added a comment.Nov 17 2023, 9:55 AM

Change 975217 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] profile::prometheus::ops: remove ORES Redis configs

https://gerrit.wikimedia.org/r/975217

gerritbot added a comment.Nov 17 2023, 9:55 AM

Change 975218 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] cloud: Remove ores-beta ATS settings

https://gerrit.wikimedia.org/r/975218

gerritbot added a comment.Nov 17 2023, 9:55 AM

Change 975219 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] admin: remove ores-admins group

https://gerrit.wikimedia.org/r/975219

gerritbot added a comment.Nov 17 2023, 9:55 AM

Change 975220 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] contactgroups: remove old team-scoring

https://gerrit.wikimedia.org/r/975220

gerritbot added a comment.Nov 17 2023, 11:03 AM

Change 975213 merged by Elukey:

[operations/puppet@production] Remove ORES roles and configs

https://gerrit.wikimedia.org/r/975213

elukey added a comment.Nov 17 2023, 11:22 AM

Updated Wikitech and Mediawiki documentation pages about ORES with a deprecation banner.

gerritbot added a comment.Nov 17 2023, 1:22 PM

Change 975267 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] profile::httpbb: remove ores_test configs

https://gerrit.wikimedia.org/r/975267

gerritbot added a comment.Nov 17 2023, 1:29 PM

Change 975267 merged by Elukey:

[operations/puppet@production] profile::httpbb: remove ores_test configs

https://gerrit.wikimedia.org/r/975267

gerritbot added a comment.Nov 17 2023, 1:30 PM

Change 975214 merged by Elukey:

[operations/puppet@production] profile::logstash: remove ORES configs

https://gerrit.wikimedia.org/r/975214

gerritbot added a comment.Nov 17 2023, 1:31 PM

Change 975215 merged by Elukey:

[operations/puppet@production] Remove ORES deployment settings

https://gerrit.wikimedia.org/r/975215

gerritbot added a comment.Nov 17 2023, 1:35 PM

Change 975216 merged by Elukey:

[operations/puppet@production] Remove ORES configs and clusters

https://gerrit.wikimedia.org/r/975216

gerritbot added a comment.Nov 17 2023, 1:36 PM

Change 975217 merged by Elukey:

[operations/puppet@production] profile::prometheus::ops: remove ORES Redis configs

https://gerrit.wikimedia.org/r/975217

gerritbot added a comment.Nov 17 2023, 1:44 PM

Change 975219 merged by Elukey:

[operations/puppet@production] admin: remove ores-admins group

https://gerrit.wikimedia.org/r/975219

gerritbot added a comment.Nov 17 2023, 1:44 PM

Change 975220 merged by Elukey:

[operations/puppet@production] contactgroups: remove old team-scoring

https://gerrit.wikimedia.org/r/975220

gerritbot added a comment.Nov 17 2023, 1:49 PM

Change 975218 merged by Elukey:

[operations/puppet@production] cloud: Remove ores-beta ATS settings

https://gerrit.wikimedia.org/r/975218

Maintenance_bot removed a project: Patch-For-Review.Nov 17 2023, 2:10 PM
gerritbot added a comment.Nov 17 2023, 2:38 PM

Change 975285 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] Clean up ores configs not used anymore

https://gerrit.wikimedia.org/r/975285

gerritbot added a project: Patch-For-Review.Nov 17 2023, 2:38 PM
elukey updated the task description. (Show Details)Nov 17 2023, 2:42 PM
elukey updated the task description. (Show Details)Nov 17 2023, 2:47 PM

@klausman everything should be done, except the work in T349632, lemme know if anything is missing, otherwise this is done.

gerritbot added a comment.Nov 17 2023, 2:48 PM

Change 975285 merged by Elukey:

[operations/puppet@production] Clean up ores configs not used anymore

https://gerrit.wikimedia.org/r/975285

klausman added a comment.Nov 17 2023, 2:50 PM
In T347278#9340974, @elukey wrote:

@klausman everything should be done, except the work in T349632, lemme know if anything is missing, otherwise this is done.

Thank you! I'll do another sweep of the assorted SRE repos (puppet, dns, private, ...) and close the ticket when done

gerritbot added a comment.Nov 20 2023, 12:09 PM

Change 975780 had a related patch set uploaded (by Klausman; author: Klausman):

[operations/puppet@production] Clean up additional ORES leftovers

https://gerrit.wikimedia.org/r/975780

gerritbot added a comment.Nov 20 2023, 3:28 PM

Change 975780 merged by Klausman:

[operations/puppet@production] Clean up additional ORES leftovers

https://gerrit.wikimedia.org/r/975780

elukey added a comment.Dec 4 2023, 11:01 AM

Deleted two Redis passwords in the private puppet repo.

elukey added a comment.Dec 4 2023, 11:02 AM
In T347278#9340995, @klausman wrote:
In T347278#9340974, @elukey wrote:

@klausman everything should be done, except the work in T349632, lemme know if anything is missing, otherwise this is done.

Thank you! I'll do another sweep of the assorted SRE repos (puppet, dns, private, ...) and close the ticket when done

Ping :)

Maintenance_bot removed a project: Patch-For-Review.Dec 4 2023, 11:10 AM
klausman added a comment.Dec 4 2023, 11:20 AM

These are the remaining hits in the puppet repo.

We need to keep these:

hieradata/common/profile/kubernetes/deployment_server.yaml
336:    ores-legacy:
338:        - name: ores-legacy
341:        - name: ores-legacy-deploy

hieradata/common/profile/trafficserver/backend.yaml
135:      target: http://ores.wikimedia.org
136:      replacement: https://ores-legacy.discovery.wmnet:31443
138:      target: http://ores-legacy.wikimedia.org
139:      replacement: https://ores-legacy.discovery.wmnet:31443

hieradata/role/common/cache/text.yaml
94:  ores.wikimedia.org:
96:  ores-legacy.wikimedia.org:

modules/profile/files/sre/buster.yaml
89:role::ores:

The last one is mostly for historical accuracy/tracking.

Remainder:

modules/profile/files/toolforge/legacy_redirector.lua
365:        'order-user-by-reg', 'ordia', 'orejasbot', 'ores', 'ores-afc',
366:        'ores-demos', 'ores-support-checklist', 'orphan-groups', 'orphantalk'

These probably can go.

modules/profile/manifests/mediawiki/maintenance/growthexperiments.pp
16:        command  => '/usr/local/bin/foreachwikiindblist /srv/mediawiki/dblists/growthexperiments.dblist extensions/GrowthExperiments/maintenance/listTaskCounts.php --topictype ores --statsd --output none',

I don't know if this somehow ties into the ORES extension.

modules/profile/templates/wmcs/backy2/wmcs_backup_instances.yaml.erb
77:  ores: cloudbackup1003

This probably can go.

modules/service/manifests/uwsgi.pp
22:#   Note: this parameter will be removed onces ores.wmflabs.org stops using service::uwsgi

This one I would keep, as it breaks nothing and the parameter in question is likely a bigger patch to remove.

From the private repo (and thus also on the pm):

hieradata/role/common/deployment_server/kubernetes.yaml
102:    ores-legacy:

Need to keep this.

hieradata/role/common/ores.yaml
1:profile::ores::web::redis_password: apassword

hieradata/role/common/ores/redis.yaml
1:profile::ores::redis::password: dummypass

hieradata/role/common/scb.yaml
2:profile::ores::web::redis_password: apassword

These likely can go, as Redis is no longer used.

I will provide patches for puppet and private repo and link them to this bug.

gerritbot added a comment.Dec 4 2023, 11:28 AM

Change 979915 had a related patch set uploaded (by Klausman; author: Klausman):

[labs/private@master] hiera: clean up more ORES leftovers

https://gerrit.wikimedia.org/r/979915

gerritbot added a project: Patch-For-Review.Dec 4 2023, 11:28 AM
gerritbot added a comment.Dec 4 2023, 11:30 AM

Change 979916 had a related patch set uploaded (by Klausman; author: Klausman):

[operations/puppet@production] profiles: Remove more ORES leftovers

https://gerrit.wikimedia.org/r/979916

gerritbot added a comment.Dec 4 2023, 2:32 PM

Change 979915 merged by Klausman:

[labs/private@master] hiera: clean up more ORES leftovers

https://gerrit.wikimedia.org/r/979915

klausman mentioned this in rLPRIf78f24f25fa1: hiera: clean up more ORES leftovers.Dec 4 2023, 2:33 PM
gerritbot added a comment.Dec 4 2023, 2:43 PM

Change 979916 merged by Klausman:

[operations/puppet@production] profiles: Remove more ORES leftovers

https://gerrit.wikimedia.org/r/979916

klausman closed this task as Resolved.Dec 4 2023, 2:44 PM

We're all done here. The to-be-archived repos we'll handle in the separate ticket.

klausman moved this task from Ready To Go to 2023-2024 Q3 Done on the Machine-Learning-Team board.Dec 4 2023, 2:45 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL