Page MenuHomePhabricator
Create Task
Maniphest T336738

Refactor our existing Airflow dags to use EasyDAG & DagProperties
Open, Needs TriagePublic
Actions

Description

During the migration from Oozie to Airflow, we have progressively refined helpers and defined conventions to write dags. Nowadays, we are using EasyDAG & DagProperties.

We should refactor something like 61 files. Hopefully, each rewrite will be fast.

ack -l "from airflow import DAG" analytics/dags/ analytics_test/dags/ | wc -l
61

Related Objects
Search...

Event Timeline

Antoine_Quhen created this task.May 16 2023, 9:47 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 16 2023, 9:47 AM
Antoine_Quhen updated the task description. (Show Details)May 16 2023, 9:48 AM
Antoine_Quhen added a parent task: T336739: Post Oozie -> Airflow migration refactorings.May 16 2023, 9:52 AM
lbowmaker moved this task from Backlog to To be discussed /To be estimated on the Data Pipelines board.May 30 2023, 1:31 PM
JArguello-WMF added a project: Data Engineering and Event Platform Team.Jun 30 2023, 5:59 PM
JArguello-WMF moved this task from Data Eng Backlog to To be Estimated/To be discussed on the Data Engineering and Event Platform Team board.
xcollazo mentioned this in T343154: [opsweek] Airflow DAGs with Spark jobs should always include Spark tuning variables.Jul 31 2023, 6:45 PM
lbowmaker moved this task from To be Estimated/To be discussed to Icebox (not considered in current quarter) on the Data Engineering and Event Platform Team board.Oct 19 2023, 8:09 PM
lbowmaker edited projects, added Data-Engineering; removed Data Engineering and Event Platform Team.Nov 10 2023, 2:23 PM
lbowmaker moved this task from Incoming (new tickets) to Icebox (not considered in current quarter) on the Data-Engineering board.
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits