After the discussions with the team, we decided to refactor the way we override DAG properties for testing and other purposes.
So far it has been done with VariableProperties, however it is difficult to grasp, especially for newcomers or non-super-technical users.
This task is about creating another mechanism to allow to override the values of the DAG properties in a much easier way.
Also, the team agreed that the new way should allow/promote/enforce to put all DAG properties together at the top of the file, as opposed as having them scattered along the file.
Description
Description
Status | Subtype | Assigned | Task | ||
---|---|---|---|---|---|
Resolved | JArguello-WMF | T324485 [Airflow] Migrate Druid loading Oozie jobs - Parent task | |||
Resolved | mforns | T334096 [Airflow] Migrate unique devices Druid loading jobs | |||
Resolved | mforns | T334098 [Airflow] Improve VariableProperties |
Event Timeline
Comment Actions
An implementation proposal has been put up for review, some team-mates have already reviewed it:
https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/323