Maniphest T311824

[M] Create the section topics Airflow DAG
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	mfossati
	Jul 1 2022, 10:49 AM

Tags

Referenced Files

None

Subscribers

Description

Once T311745: [EPIC] Section topics data pipeline's Spark code is ready, create the Airflow DAG.

NOTE: Data Platform is migrating the DAGs to a new repository with different Airflow operator wrappers, see T311417: Migration of Image Suggestion Job to DE Data Pipeline. This DAG should comply with the new conventions.

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Resolved		CBogen	T275882 [EPIC] Structured sections with topical metadata
		Resolved		CBogen	T311745 [EPIC] Section topics data pipeline
		Resolved		mfossati	T311824 [M] Create the section topics Airflow DAG

Event Timeline

mfossati created this task.Jul 1 2022, 10:49 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 1 2022, 10:49 AM

mfossati added a parent task: T311745: [EPIC] Section topics data pipeline.Jul 1 2022, 10:50 AM

CBogen moved this task from Triage to Section Topics on the Structured-Data-Backlog board.Jul 18 2022, 7:17 PM

CBogen moved this task from Section Topics to Triage on the Structured-Data-Backlog board.Aug 15 2022, 4:52 PM

CBogen edited projects, added Structured-Data-Backlog (Current Work); removed Structured-Data-Backlog.

CBogen moved this task from Incoming to Ready for Estimation on the Structured-Data-Backlog (Current Work) board.

CBogen renamed this task from Create the section topics Airflow DAG to [M] Create the section topics Airflow DAG.Aug 24 2022, 4:19 PM

CBogen moved this task from Ready for Estimation to Ready for Development on the Structured-Data-Backlog (Current Work) board.Aug 24 2022, 6:13 PM

Cparle moved this task from Ready for Development to Doing on the Structured-Data-Backlog (Current Work) board.Sep 8 2022, 12:08 PM

CBogen assigned this task to Cparle.Sep 22 2022, 8:47 PM

Cparle moved this task from Doing to Code Review on the Structured-Data-Backlog (Current Work) board.Oct 25 2022, 10:04 AM

Draft merge request at https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/136

Moving to blocked cos we need to wait for the code the DAG runs to be ready before finishing

Blocked on:

Unblocking this one to expedite production deployment. T323036: [L] Exclude media topics from section topics dataset and T323597: [M] Exclude date format topics from section topics pipeline may require a minor update to the DAG code at most.

DAG merge request at https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/136
pipeline merge request at https://gitlab.wikimedia.org/repos/structured-data/section-topics/-/merge_requests/13

Pinging @xcollazo & moving to code review.
After that, let's deploy! 🚀

Merged, released & deployed! The pipeline is now shipped to Airflow. 🚢
Closing.