Page MenuHomePhabricator

[M] Create the section topics Airflow DAG
Closed, ResolvedPublic


Once T311745: [EPIC] Section topics data pipeline's Spark code is ready, create the Airflow DAG.

NOTE: Data Platform is migrating the DAGs to a new repository with different Airflow operator wrappers, see T311417: Migration of Image Suggestion Job to DE Data Pipeline. This DAG should comply with the new conventions.

Event Timeline

CBogen renamed this task from Create the section topics Airflow DAG to [M] Create the section topics Airflow DAG.Aug 24 2022, 4:19 PM

Moving to blocked cos we need to wait for the code the DAG runs to be ready before finishing

mfossati changed the task status from Open to In Progress.Jan 23 2023, 10:51 AM
mfossati claimed this task.
mfossati moved this task from Blocked to Doing on the Structured-Data-Backlog (Current Work) board.
mfossati added a subscriber: Cparle.

Unblocking this one to expedite production deployment. T323036: [L] Exclude media topics from section topics dataset and T323597: [M] Exclude date format topics from section topics pipeline may require a minor update to the DAG code at most.

Merged, released & deployed! The pipeline is now shipped to Airflow. 🚢