Page MenuHomePhabricator

Create a new airflow package for version 2.3.2
Closed, InvalidPublic3 Estimated Story Points

Description

We need a new deb package of airflow to be created.

The code and instructions for doing so are here: https://gerrit.wikimedia.org/r/plugins/gitiles/operations/debs/airflow/

However, there is a change that we would like to make in order to stop T316592: Airflow development environment broken on all stat100x machines from happening.

The change is discussed here: https://wikimedia.slack.com/archives/CSV483812/p1662477644296739

Essentially, we want to keep the conda packages that are downloaded as part of the deb when creating it, so that the environment can be cloned again without requiring an Internet connection.

Event Timeline

BTullis set the point value for this task to 3.Sep 7 2022, 4:07 PM

@BTullis to avoid workflow_utils issues that @mforns and @NOkafor-WMF have seen, can we also bump it as part of this ticket?

Relevant line to bump: https://gerrit.wikimedia.org/r/plugins/gitiles/operations/debs/airflow/+/refs/heads/debian/pip-requirements.txt#8

Latest workflow_utils available: 0.7.0.

@BTullis to avoid workflow_utils issues that @mforns and @NOkafor-WMF have seen, can we also bump it as part of this ticket?

Relevant line to bump: https://gerrit.wikimedia.org/r/plugins/gitiles/operations/debs/airflow/+/refs/heads/debian/pip-requirements.txt#8

Latest workflow_utils available: 0.7.0.

Will do! Thanks @xcollazo.

Change 832292 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/debs/airflow@debian] Create new airflow package for version 2.3.2

https://gerrit.wikimedia.org/r/832292

We had 2 repositories defining the dependencies used by our Airflow project: airflow-dags & operations/debs/airflow.
The dependency list should be the same to minimize differences between production, development, and test environments.

So I've created a conda environment YAML file in airflow-dags to be the single source.
https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/116/diffs?commit_id=8a3c0c90e9dbf4b72e2061715cc09312d6ef9882
Also, it should simplify the creation of the matching deb package with conda create --name my_build_env -f conda-export-env.yml, no more needs for requirements generation https://gerrit.wikimedia.org/r/plugins/gitiles/operations/debs/airflow/+/refs/heads/debian/build/profile.sh#167.
@BTullis it should solve your problem of dependencies conflicts and resolutions https://phabricator.wikimedia.org/P34771

@Ottomata what do you think?

@BTullis If this is validated, I can help you to update the recipe.

cc @Snwachukwu

This looks great. Thanks Antoine. How can I help at this point?

EChetty updated Other Assignee, added: BTullis; removed: Antoine_Quhen.

Change 832292 abandoned by Btullis:

[operations/debs/airflow@debian] Create new airflow package for version 2.3.2

Reason:

Superseded by gitlab-ci build of airflow

https://gerrit.wikimedia.org/r/832292

We are on Airflow 2.5.1, and we don't have the dev environment issue anymore.

Should we close this?