Page MenuHomePhabricator

Update Airflow DAGs code to make it compatible with version V2.3.4 of Airflow
Closed, ResolvedPublic5 Estimated Story Points

Description

Steps

  • First, create a new branch
  • upgrade (if possible, see about installing Datahub lineage plugin)
  • see what tests are returning, and estimate the time needed (⚠ with macros)
  • Perform the fix

(This ticket is not about deploying the new version on the cluster. It's just about updating airflow dags to the last version of Airflow.)

Additional Tasks

  • Ensure all dags have tests => children task added
  • Manually run select jobs

Event Timeline

EChetty renamed this task from Airflow upgrade to Airflow Upgrade Compatibility with V2.3 .Jun 30 2022, 6:44 PM
EChetty renamed this task from Airflow Upgrade Compatibility with V2.3 to Airflow Upgrade Compatibility with V2.3.2 .
EChetty updated the task description. (Show Details)
EChetty set the point value for this task to 5.

The airflow dags have been update to version 2.3.2. However, before this change can be merged, we'll need to perform

  1. a puppet change on the airflow.cfg file.
  2. upgrade the airflow deb used by all airflow instances.

There will be seperate tickets to track these steps above.

BTullis renamed this task from Airflow Upgrade Compatibility with V2.3.2 to Verify DAG Compatibility with version V2.3.4 of Airflow .Dec 7 2022, 11:31 AM

I have updated the title and description to add clarity to the specific purpose of this ticket.

Also please see the following note from the security team regarding the need to use version 2.3.4 as a minimum: T315580#8208647

Antoine_Quhen renamed this task from Verify DAG Compatibility with version V2.3.4 of Airflow to Update Airflow DAGs code to make it compatible with version V2.3.4 of Airflow.Jan 4 2023, 8:52 AM
Antoine_Quhen updated the task description. (Show Details)