Page MenuHomePhabricator
Create Task
Maniphest T358793

Decommission AQS 1.0
Closed, ResolvedPublic
Actions

Description

Description

Now that AQS 2.0 has run without major incident for >2 months Data Products
would be comfortable that AQS 1.0 be fully decommissioned.

Removal of this service is important as it will have significant impacts on Data-Persistence
team as it creates overhead in maintaining and will waste resources for no purpose.

There should be no traffic to AQS 1.0: https://grafana.wikimedia.org/d/2urzFwgIk/hnowlan-aqs-usage?orgId=1&from=1706540164273&to=1709132164274&viewPanel=2

Request

For Data SRE to evaluate what is required to remove AQS 1.0 from production and schedule removal.

Details

SubjectRepoBranchLines +/-
aqs: remove conftool data and envoy listeneroperations/puppetproduction+0 -5
aqs: fix puppet compilation erroroperations/puppetproduction+0 -4
aqs: Remove conftool data and service entryoperations/puppetproduction+0 -61
aqs: remove the relserver pool from hostoperations/puppetproduction+2 -4
Set state of aqs service to lvs_setupoperations/puppetproduction+1 -1
Decommission aqs recordsoperations/dnsmaster+0 -6
hieradata: set aqs to non-pagingoperations/puppetproduction+1 -0
AQS1.0: disable aqs serviceoperations/puppetproduction+4 -0
Customize query in gerrit

Related Objects

Event Timeline

WDoranWMF triaged this task as High priority.Feb 29 2024, 5:39 PM
WDoranWMF created this task.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 29 2024, 5:39 PM
WDoranWMF removed a project: Data-Platform-SRE.Feb 29 2024, 5:40 PM
WDoranWMF added projects: Data-Platform-SRE, Data-Persistence.Feb 29 2024, 5:47 PM
KOfori removed a project: Data-Persistence.Feb 29 2024, 5:51 PM
KOfori added a project: Cassandra.
Gehel edited projects, added Data-Platform-SRE (2024.03.04 - 2024.03.24); removed Data-Platform-SRE.Feb 29 2024, 7:23 PM
Gehel subscribed.Feb 29 2024, 7:30 PM

@WDoranWMF, just to make sure I understand: this is about decommissioning the AQS servers (/^aqs10(1[0-9]|2[0-1])\.eqiad\./ and /^aqs200[1-9]|aqs201[0-2]\.codfw\./), including some cleanup of puppet code, with the service already not in use, but still running. There is also Cassandra running on those hosts, but that can also be trashed, with no need to preserve any of the data?

WDoranWMF added a subscriber: KOfori.Feb 29 2024, 7:33 PM

@Gehel Yes, though I think Data-Persistence can help with the Cassandra part if needed. Is that fair, @KOfori ?

Eevans subscribed.Feb 29 2024, 10:02 PM

This would be great; Long term, decoupling the service from the storage cluster is going to simply maintenance a great deal. In the nearer term, this will unblock upgrading to Debian Bookworm (the transition to Bullseye was already problematic).

Eevans awarded a token.Feb 29 2024, 10:03 PM
Eevans added a comment.Feb 29 2024, 10:07 PM
In T358793#9588655, @Gehel wrote:

@WDoranWMF, just to make sure I understand: this is about decommissioning the AQS servers (/^aqs10(1[0-9]|2[0-1])\.eqiad\./ and /^aqs200[1-9]|aqs201[0-2]\.codfw\./), including some cleanup of puppet code, with the service already not in use, but still running. There is also Cassandra running on those hosts, but that can also be trashed, with no need to preserve any of the data?

Nooooo, we need to leave the cluster in place, and de-deploy (un-deploy?) the AQS service only.

Please, no one decommission the servers/Cassandra :)

WDoranWMF added a project: Data-Persistence.Mar 7 2024, 5:10 PM
VirginiaPoundstone moved this task from Radar (other teams) to To be discussed on the Data Products board.Mar 18 2024, 8:33 PM
Milimetric mentioned this in T360522: aqs endpoint health alerting about mismatched check.Mar 20 2024, 2:23 PM
gerritbot added a comment.Mar 20 2024, 2:40 PM

Change 1013063 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/puppet@production] AQS1.0: disable aqs service

https://gerrit.wikimedia.org/r/1013063

gerritbot added a project: Patch-For-Review.Mar 20 2024, 2:40 PM
gerritbot added a comment.Mar 21 2024, 10:31 AM

Change 1013063 merged by Brouberol:

[operations/puppet@production] AQS1.0: disable aqs service

https://gerrit.wikimedia.org/r/1013063

Maintenance_bot removed a project: Patch-For-Review.Mar 21 2024, 11:30 AM
gerritbot added a comment.Mar 21 2024, 2:46 PM

Change #1013325 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/puppet@production] hieradata: set aqs to non-paging

https://gerrit.wikimedia.org/r/1013325

gerritbot added a project: Patch-For-Review.Mar 21 2024, 2:46 PM

Change #1013325 merged by Filippo Giunchedi:

[operations/puppet@production] hieradata: set aqs to non-paging

https://gerrit.wikimedia.org/r/1013325

Gehel moved this task from Backlog to In Progress on the Data-Platform-SRE (2024.03.04 - 2024.03.24) board.Mar 22 2024, 8:46 AM
Gehel edited projects, added Data-Platform-SRE (2024.03.25 - 2024.04.14); removed Data-Platform-SRE (2024.03.04 - 2024.03.24).Mar 22 2024, 8:48 AM
gerritbot added a comment.Mar 22 2024, 8:58 AM

Change #1013500 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/dns@master] Decommission aqs records

https://gerrit.wikimedia.org/r/1013500

gerritbot added a comment.Mar 22 2024, 9:05 AM

Change #1013501 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/puppet@production] Decommission aqs realserver pool

https://gerrit.wikimedia.org/r/1013501

gerritbot added a comment.Mar 25 2024, 2:34 PM

Change #1013500 merged by Brouberol:

[operations/dns@master] Decommission aqs records

https://gerrit.wikimedia.org/r/1013500

gerritbot added a comment.Mar 25 2024, 2:41 PM

Change #1014023 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/puppet@production] aqs: remove the relserver pool from host

https://gerrit.wikimedia.org/r/1014023

gerritbot added a comment.Mar 25 2024, 2:46 PM

Change #1013501 merged by Brouberol:

[operations/puppet@production] Set state of aqs service to lvs_setup

https://gerrit.wikimedia.org/r/1013501

gerritbot added a comment.Mar 25 2024, 2:52 PM

Change #1014023 merged by Brouberol:

[operations/puppet@production] aqs: remove the relserver pool from host

https://gerrit.wikimedia.org/r/1014023

Stashbot added a comment.Mar 25 2024, 3:00 PM

Mentioned in SAL (#wikimedia-operations) [2024-03-25T15:00:31Z] <brouberol> restarting pybal on lvs2014.codfw.wmnet - T358793

Stashbot added a comment.Mar 25 2024, 3:13 PM

Mentioned in SAL (#wikimedia-operations) [2024-03-25T15:13:06Z] <brouberol> restarting pybal on lvs2013.codfw.wmnet - T358793

Stashbot added a comment.Mar 25 2024, 4:05 PM

Mentioned in SAL (#wikimedia-operations) [2024-03-25T16:05:54Z] <brouberol> restarting pybal on lvs1020.eqiad.wmnet - T358793

Stashbot added a comment.Mar 25 2024, 4:12 PM

Mentioned in SAL (#wikimedia-operations) [2024-03-25T16:12:27Z] <brouberol> restarting pybal on lvs1019.eqiad.wmnet - T358793

gerritbot added a comment.Mar 25 2024, 4:29 PM

Change #1014042 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/puppet@production] aqs: Remove conftool data and service entry

https://gerrit.wikimedia.org/r/1014042

gerritbot added a comment.Mar 25 2024, 6:56 PM

Change #1014100 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/puppet@production] aqs: fix puppet compilation error

https://gerrit.wikimedia.org/r/1014100

gerritbot added a comment.Mar 25 2024, 6:58 PM

Change #1014042 merged by Brouberol:

[operations/puppet@production] aqs: Remove conftool data and service entry

https://gerrit.wikimedia.org/r/1014042

gerritbot added a comment.Mar 25 2024, 7:17 PM

Change #1014100 merged by Brouberol:

[operations/puppet@production] aqs: fix puppet compilation error

https://gerrit.wikimedia.org/r/1014100

brouberol subscribed.Mar 25 2024, 7:27 PM

I've removed the VIP from all AQS hosts. All there is to do now is to remove the VIP from netbox.

Stashbot added a comment.Mar 25 2024, 7:28 PM

Mentioned in SAL (#wikimedia-operations) [2024-03-25T19:28:32Z] <brouberol> removing VIP from AQS hosts - T358793

Stashbot added a comment.Mar 26 2024, 8:18 AM

Mentioned in SAL (#wikimedia-operations) [2024-03-26T08:18:45Z] <brouberol> deleting AQS codfw VIP (10.2.1.12/32) from Netbox - T358793

Stashbot added a comment.Mar 26 2024, 8:19 AM

Mentioned in SAL (#wikimedia-operations) [2024-03-26T08:19:32Z] <brouberol> deleting AQS eqiad VIP (10.2.2.12/32) from Netbox - T358793

brouberol closed this task as Resolved.Mar 26 2024, 8:20 AM
brouberol claimed this task.
gerritbot added a comment.Mar 26 2024, 9:50 AM

Change #1014441 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/puppet@production] aqs: remove conftool data and envoy listener

https://gerrit.wikimedia.org/r/1014441

gerritbot added a comment.Mar 26 2024, 9:56 AM

Change #1014441 merged by Brouberol:

[operations/puppet@production] aqs: remove conftool data and envoy listener

https://gerrit.wikimedia.org/r/1014441

brouberol moved this task from Backlog to Done on the Data-Platform-SRE (2024.03.25 - 2024.04.14) board.Mar 26 2024, 10:40 AM
WDoranWMF added a comment.Mar 26 2024, 11:57 AM

Amazing, thank you @brouberol! Do you want to have the joy of resolving the task?

brouberol added a comment.Mar 26 2024, 12:00 PM

My pleasure :) I've already set it to Resolved 👍

KOfori awarded a token.Mar 26 2024, 12:58 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL