[L] Verify all Airflow DAGs after migration to Kubernetes
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	mfossati
	Apr 2 2025, 3:17 PM

Description

T380618: Migrate the airflow-platform-eng database to Kubernetes and T380624: Migrate the airflow-platform-eng scheduler to Kubernetes moved the scheduler and all tasks to Kubernetes.

Tasks

replace BashOperators that run hive (e.g., this one) with SparkSqlOperators or SparkSubmitOperators. See also here
check any DAG task that fires API requests (action API, eventstreams, etc.) because we might need to add an envoy listener for relevant DAGs
- cassandra > check_data_and_push_metrics
- section_topics > fetch_qids_for_all_points_in_time & fetch_qids_for_media_outlets
- section_titles_denylist > gather_section_titles_denylist
- verify check_bad_parsing > check_bad_parsing's output, as action API requests failures are handled

Details

Related Changes in GitLab:

	Title	Reference	Author	Source Branch	Dest Branch
	Update image suggestions DAGs after migration to Kubernetes	repos/data-engineering/airflow-dags!1215	mfossati	T390880	main

Customize query in GitLab

Related Objects
Search...

Status	Assigned	Task
Resolved	brouberol	T362788 Migrate Airflow to the dse-k8s cluster
Resolved	brouberol	T364389 Migrate the airflow scheduler components to Kubernetes
Resolved	• Stevemunene	T380624 Migrate the airflow-platform-eng scheduler to Kubernetes
Open	None	T340437 [EPIC] Image suggestions data pipelines maintenance
Resolved	mfossati	T390880 [L] Verify all Airflow DAGs after migration to Kubernetes

Event Timeline

mfossati created this task.Apr 2 2025, 3:17 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 2 2025, 3:17 PM

mfossati moved this task from Incoming to Ready for Estimation on the Structured-Data-Backlog (Current Work) board.Apr 2 2025, 3:18 PM

mfossati added a parent task: T340437: [EPIC] Image suggestions data pipelines maintenance .Apr 2 2025, 3:20 PM

BTullis subscribed.Apr 2 2025, 4:19 PM

MarkTraceur renamed this task from Verify all Airflow DAGs after migration to Kubernetes to [L] Verify all Airflow DAGs after migration to Kubernetes.Apr 2 2025, 4:34 PM

MarkTraceur moved this task from Ready for Estimation to Ready for Development on the Structured-Data-Backlog (Current Work) board.

• Stevemunene added a parent task: T380624: Migrate the airflow-platform-eng scheduler to Kubernetes.Apr 3 2025, 8:07 AM

• Stevemunene mentioned this in T380624: Migrate the airflow-platform-eng scheduler to Kubernetes.

• Stevemunene updated the task description. (Show Details)

mfossati changed the task status from Open to In Progress.Apr 3 2025, 9:08 AM

mfossati claimed this task.

mfossati moved this task from Ready for Development to Doing on the Structured-Data-Backlog (Current Work) board.

mfossati opened https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/1215

Update image suggestions DAGs after migration to Kubernetes

mfossati updated the task description. (Show Details)Apr 4 2025, 4:35 PM

mfossati updated the task description. (Show Details)Apr 7 2025, 10:09 AM

mfossati merged https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/1215