Migrate the airflow-analytics-product scheduler to Kubernetes
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	brouberol
	Nov 22 2024, 3:54 PM

Details

Related Changes in Gerrit:

Subject	Repo	Branch	Lines +/-
analytics-product: enable traffic to analytics-web listener	operations/deployment-charts	master	+4 -0
envoy: add the analytics-web service to the mesh	operations/puppet	production	+7 -0
Define the analytics-web service	operations/puppet	production	+18 -0
an-web: enable traffic to port 8443 from the dse-k8s kubernetes cluster	operations/puppet	production	+1 -0
airflow-analytics-product: disable and remove the airflow systemd services	operations/puppet	production	+1 -0
airflow-analytics-product: add missing database value	operations/deployment-charts	master	+9 -0
airflow-analytics-product: migrate the scheduler and the DB to Kubernetes	operations/deployment-charts	master	+17 -37

Customize query in gerrit

Related Changes in GitLab:

	Title	Reference	Author	Source Branch	Dest Branch
	analytics_product: connect to analytics.wikimedia.org via the service mesh	repos/data-engineering/airflow-dags!1092	brouberol	T380623	main

Customize query in GitLab

Related Objects
Search...

Status	Assigned	Task
Resolved	brouberol	T362788 Migrate Airflow to the dse-k8s cluster
Resolved	brouberol	T364389 Migrate the airflow scheduler components to Kubernetes
Resolved	brouberol	T380623 Migrate the airflow-analytics-product scheduler to Kubernetes

Event Timeline

brouberol created this task.Nov 22 2024, 3:54 PM

Gehel triaged this task as High priority.Nov 25 2024, 1:33 PM

Gehel moved this task from Incoming to Scratch on the Data-Platform-SRE board.Jan 8 2025, 8:39 AM

Gehel edited projects, added Data-Platform-SRE (2024.11.30 - 2024.12.20); removed Data-Platform-SRE.

Gehel edited projects, added Data-Platform-SRE (2025.01.11 - 2025.01.31); removed Data-Platform-SRE (2024.11.30 - 2024.12.20).Jan 10 2025, 10:39 AM

Gehel edited projects, added Data-Platform-SRE (2025.02.10 - 2025.02.28); removed Data-Platform-SRE (2025.01.11 - 2025.01.31).Feb 11 2025, 10:05 AM

brouberol@stat1008:~$ s3cmd --access_key=$access_key --secret_key=$secret_key --host=rgw.eqiad.dpe.anycast.wmnet --region=dpe --host-bucket=no mb s3://logs.airflow-analytics-product.dse-k8s-eqiad
Bucket 's3://logs.airflow-analytics-product.dse-k8s-eqiad/' created

Change #1122591 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/deployment-charts@master] airflow-analytics-product: migrate the scheduler and the DB to Kubernetes

https://gerrit.wikimedia.org/r/1122591

Change #1122592 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/puppet@production] airflow-analytics-product: disable and remove the airflow systemd services

https://gerrit.wikimedia.org/r/1122592

Change #1122591 merged by Brouberol:

[operations/deployment-charts@master] airflow-analytics-product: migrate the scheduler and the DB to Kubernetes

https://gerrit.wikimedia.org/r/1122591

Change #1122944 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/deployment-charts@master] airflow-analytics-product: add missing database value

https://gerrit.wikimedia.org/r/1122944

Change #1122944 merged by Brouberol:

[operations/deployment-charts@master] airflow-analytics-product: add missing database value

https://gerrit.wikimedia.org/r/1122944

Change #1122592 merged by Brouberol:

[operations/puppet@production] airflow-analytics-product: disable and remove the airflow systemd services

https://gerrit.wikimedia.org/r/1122592

The flagged_revisions_pending_hourly DAG executed on Kubernetes from start to end without any issue. Given that this is the only non-daily/monthly DAG, I'm going to move this ticket in our Blocked/Waiting column until we get more results tomorrow. Onwards!

Maintenance_bot removed a project: Patch-For-Review.Feb 26 2025, 2:30 PM

Change #1123289 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/puppet@production] Define the analytics-web service

https://gerrit.wikimedia.org/r/1123289

gerritbot added a project: Patch-For-Review.Feb 27 2025, 8:47 AM

Change #1123290 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/puppet@production] envoy: add the analytics-web service to the mesh

https://gerrit.wikimedia.org/r/1123290

Change #1123300 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/puppet@production] an-web: enable traffic to port 8443 from the dse-k8s kubeernetes cluster

https://gerrit.wikimedia.org/r/1123300

Change #1123300 merged by Brouberol:

[operations/puppet@production] an-web: enable traffic to port 8443 from the dse-k8s kubernetes cluster

https://gerrit.wikimedia.org/r/1123300

brouberol opened https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/1092

analytics_product: connect to analytics.wikimedia.org via the service mesh

Change #1123308 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/deployment-charts@master] analytics-product: enable traffic to analytics-web listener

https://gerrit.wikimedia.org/r/1123308

Change #1123289 merged by Brouberol:

[operations/puppet@production] Define the analytics-web service

https://gerrit.wikimedia.org/r/1123289

Change #1123290 merged by Brouberol:

[operations/puppet@production] envoy: add the analytics-web service to the mesh

https://gerrit.wikimedia.org/r/1123290

Change #1123308 merged by Brouberol:

[operations/deployment-charts@master] analytics-product: enable traffic to analytics-web listener

https://gerrit.wikimedia.org/r/1123308

The automoderator_monitoring_snapshot_daily DAG failed last night, due to https://analytics.wikimedia.org not being reachable from the airflow instance. We're solving this issue ny making the service reachable through the service mesh.