Page MenuHomePhabricator

Make sure we install python 3.10 in our airflow worker image
Closed, ResolvedPublic

Description

We recently had to downgrade our airflow worker Docker image from Bookworm to Bullseye, as we've not (yet?) been successful at building bigtop 1.5 for Bookworm (cf T378954). As a result, the system python version got downgraded from 3.11 to 3.9.

Both are wrong in any case, as our current airflow conda environment pins python to 3.10

Let's make sure we decorrelate the worker python version from the Debian version, by installing python using something like pyenv instead of using the system python,

Event Timeline

brouberol opened https://gitlab.wikimedia.org/repos/data-engineering/airflow/-/merge_requests/29

Draft: Install python 3.10 with pyenv and install airflow/dependencies with it

brouberol merged https://gitlab.wikimedia.org/repos/data-engineering/airflow/-/merge_requests/29

Install python 3.10 with pyenv and install airflow/dependencies with it

brouberol opened https://gitlab.wikimedia.org/repos/data-engineering/airflow/-/merge_requests/30

Mount the pyenv directory in the same location in the build and base stages

brouberol merged https://gitlab.wikimedia.org/repos/data-engineering/airflow/-/merge_requests/30

Mount the pyenv directory in the same location in the build and base stages

BTullis triaged this task as High priority.Nov 8 2024, 10:56 AM

Change #1088561 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/deployment-charts@master] airflow: upgrade docker image to run airflow with python 3.10

https://gerrit.wikimedia.org/r/1088561

Change #1088561 merged by Brouberol:

[operations/deployment-charts@master] airflow: upgrade docker image to run airflow with python 3.10

https://gerrit.wikimedia.org/r/1088561