Page MenuHomePhabricator

airflow-devenv fails to create when --name is long-ish
Closed, ResolvedPublic

Description

xcollazo@deploy1003:~$ airflow-devenv create -d main -b multiple-fixes-for-mw-reconcile -n mw-reconcile-fixes
Airflow development environment provisioning parameters:
- instance name: mw-reconcile-fixes
- pulled git branch: multiple-fixes-for-mw-reconcile
- airflow-dags folder: main

Installation progress:
- Creating PG database mw_reconcile_fixes ✅
- Installing airflow dev environment mw-reconcile-fixeshelm install mw-reconcile-fixes /srv/deployment-charts/charts/airflow -f /etc/helmfile-defaults/general-dse-k8s-eqiad.yaml -f /etc/helmfile-defaults/private/dse-k8s_services/airflow-dev/dse-k8s-eqiad.yaml -f /srv/deployment-charts/helmfile.d/dse-k8s-services/_airflow_common_/values-dse-k8s-eqiad.yaml -f /srv/deployment-charts/helmfile.d/dse-k8s-services/_airflow_common_/values-analytics-production.yaml -f /srv/deployment-charts/helmfile.d/dse-k8s-services/airflow-main/values-production.yaml -f /srv/deployment-charts/helmfile.d/dse-k8s-services/_airflow_common_/values-dev.yaml --set devenv.db.name=mw_reconcile_fixes --set gitsync.ref=multiple-fixes-for-mw-reconcile --set config.airflow.config.kerberos.principal=xcollazo@WIKIMEDIA --set environmentName=dse-k8s-eqiad --set pgServiceName=postgresql-airflow-dev --set-json config.connections.datahub_superset_ingestion.login="[REDACTED]" --set-json config.connections.datahub_superset_ingestion.password="[REDACTED]"
Status: FAIL WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /etc/kubernetes/airflow-dev-deploy-dse-k8s-eqiad.config
W0924 13:16:19.790631 3877833 warnings.go:70] spec.template.spec.initContainers[0].env[5].name: duplicate name "POOLER_NAME"
W0924 13:16:19.790640 3877833 warnings.go:70] spec.template.spec.containers[0].env[5].name: duplicate name "POOLER_NAME"
W0924 13:16:19.797561 3877833 warnings.go:70] spec.template.spec.initContainers[0].env[5].name: duplicate name "POOLER_NAME"
W0924 13:16:19.797569 3877833 warnings.go:70] spec.template.spec.containers[0].env[5].name: duplicate name "POOLER_NAME"
W0924 13:16:19.798300 3877833 warnings.go:70] spec.template.spec.containers[0].env[5].name: duplicate name "POOLER_NAME"
Error: INSTALLATION FAILED: 1 error occurred:
	* Deployment.apps "airflow-mw-reconcile-fixes-hadoop-shell" is invalid: [spec.template.spec.volumes[3].name: Invalid value: "airflow-mw-reconcile-fixes-worker-extra-configuration-etc-refinery": must be no more than 63 characters, spec.template.spec.containers[0].volumeMounts[3].name: Not found: "airflow-mw-reconcile-fixes-worker-extra-configuration-etc-refinery"]
`

Slack thread.

Project work is implemented but the deployment is actively blocked by T408365

Event Timeline

xcollazo renamed this task from airflow-devenv fails to create when branch name is long-ish to airflow-devenv fails to create when --name is long-ish.Sep 25 2025, 2:07 PM

Change #1191072 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/deployment-charts@master] airflow: automatically figure out some values to reduce release config size

https://gerrit.wikimedia.org/r/1191072

brouberol changed the task status from Open to In Progress.Sep 29 2025, 10:46 AM
brouberol claimed this task.

Change #1191072 merged by jenkins-bot:

[operations/deployment-charts@master] airflow: automatically figure out some values to reduce release config size

https://gerrit.wikimedia.org/r/1191072

@Stevemunene Now that the airflow chart UX improvements have landed (and that the worker extra config files Configmap name length was reduced), would you mind adding a validator on top of the airflow-devenv create --name flag? Its max length should be 25.

Thanks!

Getting an error setting up the build on the buid2002 host which I am looking into

Reading package lists...
E: The repository 'http://mirrors.wikimedia.org/debian bullseye-backports Release' does not have a Release file.
I: unmounting dev/ptmx filesystem
I: unmounting dev/pts filesystem
I: unmounting dev/shm filesystem
I: unmounting proc filesystem
I: unmounting sys filesystem
I: Cleaning COW directory
I: forking: rm -rf /var/cache/pbuilder/build/cow.1148396
stevemunene@build2002:~/airflow-devenv$

The error above was due to T383557, removed the backports option but I amrunning into this error now

E: pbuilder-satisfydepends failed.

Some observations,
When installing using BACKPORTS=yes DIST=bullseye-wikimedia pdebuild, despite the fact that there are no mirror etc. E: The repository 'http://mirrors.wikimedia.org/debian bullseye-backports Release' does not have a Release file. all the required packages are built.

However when using DIST=bullseye-wikimedia pdebuild, we get an error that pbuilder-satisfydepends-dummy cannot find the core python3-poetry-core and pybuild-plugin-pyproject then fails

The following packages have unmet dependencies:
 pbuilder-satisfydepends-dummy : Depends: python3-poetry-core which is a virtual package and is not provided by any available package

                                 Depends: pybuild-plugin-pyproject which is a virtual package and is not provided by any available package

Unable to resolve dependencies!  Giving up...

Tried building with bookworm packages from https://packages.debian.org/search?keywords=pybuild-plugin-pyproject and https://packages.debian.org/bookworm/python3-poetry-core but the end result is still the same.
both APT_USE_BUILT=yes BUILDRESULT=/home/stevemunene/pydep/pybuild-plugin-pyproject_5.20230130+deb12u1_all.deb BUILDRESULT=/home/stevemunene/pydep/python3-poetry-core_1.4.0-4_all.deb DIST=bullseye pdebuild and APT_USE_BUILT=yes BUILDRESULT=/home/stevemunene/pydep DIST=bullseye pdebuild

The following packages have unmet dependencies:
 pbuilder-satisfydepends-dummy : Depends: python3-poetry-core which is a virtual package and is not provided by any available package

                                 Depends: pybuild-plugin-pyproject which is a virtual package and is not provided by any available package

Unable to resolve dependencies!  Giving up...

It seems our package does not support bookworm, giving it another try with trixie