In order to prepare for upgrade planning, I'll run minimal airflow-devenv enviroment with minimal possible dags (or no dags)
Quiestions:
Acceptance criteria:
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Restricted Task | |||||
| Resolved | BTullis | T423243 Upgrade Airflow to 2.11.2 | |||
| In Progress | atsuko | T423248 Upgrade to Airflow 3.3.x | |||
| Resolved | atsuko | T432993 Package and explore Airflow 3.3.0 in airflow-devenv |
UPD:
airflow-devenv is pulling the configs manually, so it is not needed.
ENVIRONMENT_CONFIG_FILE = (
f"{AIRFLOW_COMMON_HELMFILE_DIRECTORY}/values-{ENVIRONMENT}.yaml"
)
cmd = [
"helm",
"install",
...
"-f",
# airflow values common to all instances deployed in the current environment
ENVIRONMENT_CONFIG_FILE,Version of image is pinned in helmfile, while the chart is pulling the latest version.
app: version: &base-image-version airflow-2.11.2-py3.11-2026-06-10-131616-6b0fbe1f9b58505d433c58641979ec063ddb15ee@sha256:18e5bf4bbd07aa0a8eb9a93d7f1dc52640f16d7bc973b3cca6398ea2c868702d executor_pod_image_version: *base-image-version
Chart need to be modified to use a tag:
app: image: &base-image repos/data-engineering/airflow-dags version: &base-image-version latest # we use latest everywhere in the defaults. executor_pod_image: *base-image executor_pod_image_version: *base-image-version
Change #1315759 had a related patch set uploaded (by Atsuko; author: Atsuko):
[operations/deployment-charts@master] charts/airflow: pinning image version in chart
Change #1315759 abandoned by Atsuko:
[operations/deployment-charts@master] charts/airflow: pinning image version in chart
Reason:
Turns out we are already using the config in `airflow-devenv`
atsuko opened https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/2422
Draft: Building Airflow 3.3.0 image
Blubber is using airflow-dags:latest tag
build-test-image: base: docker-registry.wikimedia.org/repos/data-engineering/airflow-dags:latest
It should be fine as long as I build with explicit tag
docker buildx build -f docker/blubber.yaml --load --target build-prod-image -t docker-registry.wikimedia.org/repos/data-engineering/airflow-dags:airflow-3.3.0-py3.11 .
Setting up the infra.
atsuko opened https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/2433
Draft: Build Airflow 3.3.0 images on trusted runner
atsuko merged https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/2433
Build Airflow 3.3.0 images on trusted runner
atsuko merged https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/2422
Building Airflow 3.3.0 image
Built and published docker-registry.wikimedia.org/repos/data-engineering/airflow-dags:airflow-3.3.0-py3.11-2026-07-27-144747-017fb2ebfb3d51e3b2ddc7769d6207a76e3085a8@sha256:d6f071cb28188de4804e6e8317de1b8cb1628a22fbdeb461cae3c66f56695e3e
$ VER=airflow-3.3.0-py3.11-2026-07-27-144747-017fb2ebfb3d51e3b2ddc7769d6207a76e3085a8 $ airflow-devenv create -d test_k8s --branch main_airflow_3.3 --set app.version=${VER},app.executor_pod_image_version=${VER},config.airflow.config.kubernetes_executor.worker_container_tag=${VER}
atsuko opened https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/2438
Building Airflow 3.3.0 image
atsuko merged https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/2438
Building Airflow 3.3.0 image
Change #1318719 had a related patch set uploaded (by Atsuko; author: Atsuko):
[operations/deployment-charts@master] airflow: support for v3.3.0
Minimal experimental airflow 3.3 setup
$ mkdir deployment-charts && cd deployment-charts && git init $ git fetch https://gerrit.wikimedia.org/r/operations/deployment-charts refs/changes/19/1318719/1 && git checkout FETCH_HEAD $ VER=airflow-3.3.0-py3.11-2026-07-29-110048-56a2261c888e8687c75ea9aaa4e2ab6dfdce33f8 $ airflow-devenv create -d test_k8s --branch main_airflow_3.3 --set app.version=${VER},app.executor_pod_image_version=${VER},config.airflow.config.kubernetes_executor.worker_container_tag=${VER} --charts-folder `pwd`/charts --dags-files no-files
Todo:
atsuko updated https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/2445
Building Airflow 3.3.0 image
atsuko merged https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/2445
Building Airflow 3.3.0 image
We've got working image
2026-07-29 11:15:17,785 Target 'build-prod-image' image published to 'docker-registry.wikimedia.org/repos/data-engineering/airflow-dags:airflow-3.3.0-py3.11-2026-07-29-110048-56a2261c888e8687c75ea9aaa4e2ab6dfdce33f8@sha256:0bdaf26e8882a3a33b02226265258668f60e61d4ffbe2e1b38164a2385503063'
Change #1318719 merged by jenkins-bot:
[operations/deployment-charts@master] airflow: support for v3.3.0