Page MenuHomePhabricator
Create Task
Maniphest T389172

Decommission airflow-analytics
Closed, ResolvedPublic
Actions

Description

Are we're going to migrate the analytics DAGs to airflow-main, we can safely decommission airflow-analytics.wikimedia.org, as it's not being used.

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
Remove the airflow-analytics kubeconfigsoperations/puppetproduction+0 -8
Remove the airflow-analytics namespaceoperations/deployment-chartsmaster+0 -4
Remove the airflow-analytics deployemnt helmfile and valuesoperations/deployment-chartsmaster+0 -130
Remove the airflow-analytics namespace from the operators tenant ns listoperations/deployment-chartsmaster+0 -3
Remove airflow-analytics from the IDP configurationoperations/puppetproduction+0 -8
Remove airflow-analytics from the ATS and cache configurationoperations/puppetproduction+0 -5
Drop airflow-analytics DNS recordsoperations/dnsmaster+0 -3
Customize query in gerrit

Related Objects
Search...

Event Timeline

brouberol created this task.Mar 18 2025, 8:05 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 18 2025, 8:05 AM
brouberol added a parent task: T386282: Migrate analytics Airflow DAGs to k8s Airflow deployment.Mar 18 2025, 8:06 AM
gerritbot added a comment.Mar 18 2025, 8:09 AM

Change #1128785 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/dns@master] Drop airflow-analytics DNS records

https://gerrit.wikimedia.org/r/1128785

gerritbot added a project: Patch-For-Review.Mar 18 2025, 8:09 AM
gerritbot added a comment.Mar 18 2025, 8:11 AM

Change #1128786 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/puppet@production] Remove airflow-analytics from the ATS and cache configuration

https://gerrit.wikimedia.org/r/1128786

gerritbot added a comment.Mar 18 2025, 8:11 AM

Change #1128787 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/puppet@production] Remove airflow-analytics from the IDP configuration

https://gerrit.wikimedia.org/r/1128787

gerritbot added a comment.Mar 18 2025, 8:11 AM

Change #1128788 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/puppet@production] Remove the airflow-analytics kubeconfigs

https://gerrit.wikimedia.org/r/1128788

brouberol moved this task from Backlog - project to Needs Review on the Data-Platform-SRE (2025.03.01 - 2025.03.21) board.Mar 18 2025, 8:13 AM
gerritbot added a comment.Mar 18 2025, 8:14 AM

Change #1128789 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/deployment-charts@master] Remove the airflow-analytics namespace from the operators tenant ns list

https://gerrit.wikimedia.org/r/1128789

gerritbot added a comment.Mar 18 2025, 8:14 AM

Change #1128790 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/deployment-charts@master] Remove the airflow-analytics deployemnt helmfile and values

https://gerrit.wikimedia.org/r/1128790

gerritbot added a comment.Mar 18 2025, 8:14 AM

Change #1128791 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/deployment-charts@master] Remove the airflow-analytics namespace

https://gerrit.wikimedia.org/r/1128791

gerritbot added a comment.Mar 18 2025, 8:30 AM

Change #1128785 merged by Brouberol:

[operations/dns@master] Drop airflow-analytics DNS records

https://gerrit.wikimedia.org/r/1128785

gerritbot added a comment.Mar 18 2025, 8:32 AM

Change #1128786 merged by Brouberol:

[operations/puppet@production] Remove airflow-analytics from the ATS and cache configuration

https://gerrit.wikimedia.org/r/1128786

gerritbot added a comment.Mar 18 2025, 9:18 AM

Change #1128787 merged by Brouberol:

[operations/puppet@production] Remove airflow-analytics from the IDP configuration

https://gerrit.wikimedia.org/r/1128787

gerritbot added a comment.Mar 18 2025, 9:29 AM

Change #1128789 merged by jenkins-bot:

[operations/deployment-charts@master] Remove the airflow-analytics namespace from the operators tenant ns list

https://gerrit.wikimedia.org/r/1128789

gerritbot added a comment.Mar 18 2025, 9:53 AM

Change #1128790 merged by Brouberol:

[operations/deployment-charts@master] Remove the airflow-analytics deployemnt helmfile and values

https://gerrit.wikimedia.org/r/1128790

gerritbot added a comment.Mar 18 2025, 9:54 AM

Change #1128791 merged by Brouberol:

[operations/deployment-charts@master] Remove the airflow-analytics namespace

https://gerrit.wikimedia.org/r/1128791

gerritbot added a comment.Mar 18 2025, 9:54 AM

Change #1128788 merged by Brouberol:

[operations/puppet@production] Remove the airflow-analytics kubeconfigs

https://gerrit.wikimedia.org/r/1128788

brouberol closed this task as Resolved.Mar 18 2025, 10:07 AM
brouberol moved this task from Needs Review to Done on the Data-Platform-SRE (2025.03.01 - 2025.03.21) board.

All done!

brouberol reopened this task as Open.Mar 18 2025, 10:23 AM

Actually, let's reopen, as I still need to delete the buckets and the S3 users.

brouberol added a comment.Mar 18 2025, 10:29 AM
brouberol@stat1008:~$ s3cmd --access_key=$access_key --secret_key=$secret_key --host=rgw.eqiad.dpe.anycast.wmnet --region=dpe --host-bucket=no rb s3://postgresql-airflow-analytics.dse-k8s-eqiad/
Bucket 's3://postgresql-airflow-analytics.dse-k8s-eqiad/' removed
brouberol@stat1008:~$ s3cmd --access_key=$access_key --secret_key=$secret_key --host=rgw.eqiad.dpe.anycast.wmnet --region=dpe --host-bucket=no rb --recursive s3://logs.airflow-analytics.dse-k8s-eqiad
WARNING: Bucket is not empty. Removing all the objects from it first. This may take some time...
....
Maintenance_bot removed a project: Patch-For-Review.Mar 18 2025, 10:30 AM
brouberol added a comment.Mar 18 2025, 10:48 AM

Switching the last command to the much faster

brouberol@an-launcher1002:~$ rclone --s3-access-key-id $access_key --s3-secret-access-key $secret_key purge s3dpe://logs.airflow-analytics.dse-k8s-eqiad
dr0ptp4kt subscribed.Mar 18 2025, 11:14 AM
brouberol added a comment.Mar 18 2025, 1:16 PM

Screenshot 2025-03-18 at 14.15.50.png (2×2 px, 426 KB)

brouberol closed this task as Resolved.Mar 18 2025, 1:17 PM
brouberol@cephosd1001:~$ sudo radosgw-admin user list
[
    "airflow-research",
    "mw-content-history-reconcile-enrich",
    "airflow-search",
    "btullis",
    "postgresql-airflow-ml",
    "airflow-main",
    "airflow-analytics-product",
    "airflow-ml",
    "research",
    "airflow-wmde",
    "airflow-analytics-test",
    "postgresql-airflow-wmde",
    "mw-content-history-reconcile-enrich-next",
    "postgresql-airflow-analytics",
    "postgresql-test-k8s",
    "airflow-analytics",
    "postgresql-airflow-analytics-product",
    "postgresql-airflow-main",
    "postgresql-airflow-analytics-test",
    "postgresql-airflow-platform-eng",
    "airflow-test-k8s",
    "postgresql-airflow-search",
    "airflow-platform-eng"
]
brouberol@cephosd1001:~$ sudo radosgw-admin user rm --uid airflow-analytics
brouberol@cephosd1001:~$ sudo radosgw-admin user rm --uid postgresql-airflow-analytics
brouberol@cephosd1001:~$ sudo radosgw-admin user list
[
    "airflow-research",
    "mw-content-history-reconcile-enrich",
    "airflow-search",
    "btullis",
    "postgresql-airflow-ml",
    "airflow-main",
    "airflow-analytics-product",
    "airflow-ml",
    "research",
    "airflow-wmde",
    "airflow-analytics-test",
    "postgresql-airflow-wmde",
    "mw-content-history-reconcile-enrich-next",
    "postgresql-test-k8s",
    "postgresql-airflow-analytics-product",
    "postgresql-airflow-main",
    "postgresql-airflow-analytics-test",
    "postgresql-airflow-platform-eng",
    "airflow-test-k8s",
    "postgresql-airflow-search",
    "airflow-platform-eng"
]
brouberol@cephosd1001:~$
Gehel moved this task from Done to Reported on the Data-Platform-SRE (2025.03.01 - 2025.03.21) board.Mar 21 2025, 9:38 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits