Page MenuHomePhabricator
Paste P96527

airflow-test-k8s to airflow3
ActivePublic
Actions

Authored by atsuko on Fri, Sep 25, 10:55 AM.
Project Tags
None
Referenced Files
F104774477: airflow-test-k8s to airflow3
Fri, Sep 25, 11:09 AM
F104771325: airflow-test-k8s to airflow3
Fri, Sep 25, 10:55 AM
Subscribers
None
Comparing release=production, chart=wmf-stable/airflow, namespace=airflow-test-k8s
airflow-test-k8s, airflow-config, ConfigMap (v1) has changed:
...
xcom_objectstorage_compression =
xcom_objectstorage_path = s3://s3_dpe@logs.airflow-test-k8s.dse-k8s-eqiad/xcoms
xcom_objectstorage_threshold = 262144
[core]
+ auth_manager = auth_manager.FabAuthManager
colored_console_log = False
dags_folder = /opt/airflow/dags/airflow_dags/test_k8s
default_task_retries = 5
execution_api_server_url = https://airflow-production-tls-service.airflow-test-k8s.svc.cluster.local:1025/execution/
executor = KubernetesExecutor
...
delete_worker_pods_on_failure = True
multi_namespace_mode = False
namespace = airflow-test-k8s
pod_template_file = /opt/airflow/pod_templates/kubernetes_executor_default_pod_template.yaml
worker_container_repository = docker-registry.discovery.wmnet/repos/data-engineering/airflow-dags
- worker_container_tag = airflow-2.11.2-py3.11-2026-06-10-131616-6b0fbe1f9b58505d433c58641979ec063ddb15ee@sha256:18e5bf4bbd07aa0a8eb9a93d7f1dc52640f16d7bc973b3cca6398ea2c868702d
+ worker_container_tag = airflow-3.3.1-py3.11-2026-09-23-160148-1ae06b2b0a30b6f058d913d298829f1b94caf266@sha256:2d70acfbc0c315327b5a57df4158debd28573d45dd03714341f9201b271d3064
worker_pods_creation_batch_size = 32
[logging]
colored_console_log = False
delete_local_logs = True
...
parsing_processes = 2
pool_metrics_interval = 60
scheduler_health_check_server_host = 0.0.0.0
scheduler_health_check_server_port = 8765
scheduler_health_check_threshold = 60
- standalone_dag_processor = False
+ standalone_dag_processor = True
[secrets]
backend = airflow.secrets.local_filesystem.LocalFilesystemBackend
backend_kwargs = {"connections_file_path": "/opt/airflow/secrets/connections.yaml"}
...
airflow-test-k8s, airflow-envoy, Deployment (apps) has changed:
...
release: production
routed_via: production
component: envoy
annotations:
checksum/secrets: 65333750984d5a3960b85d7927490a3baea64a4996174311b4a8fd2df39a39d0
- checksum/configuration: 0d8f274ee3fe859082a1b84082de50bef5deed076567598110ca4ef2b9c628ac
+ checksum/configuration: a5083eae5608f9124fefb280c36541330ffd0e6c330978b8210babd3efe76322
checksum/tls-config: e7022733e316206a4aa5d56a48d4bb1574af43edc1f86be5d18704c479b1aedd
envoyproxy.io/scrape: "true"
envoyproxy.io/port: "1667"
spec:
containers:
...
airflow-test-k8s, airflow-gitsync, Deployment (apps) has changed:
...
release: production
routed_via: production
component: gitsync
annotations:
checksum/secrets: 65333750984d5a3960b85d7927490a3baea64a4996174311b4a8fd2df39a39d0
- checksum/configuration: 0d8f274ee3fe859082a1b84082de50bef5deed076567598110ca4ef2b9c628ac
+ checksum/configuration: a5083eae5608f9124fefb280c36541330ffd0e6c330978b8210babd3efe76322
spec:
securityContext:
fsGroup: 900
containers:
- name: airflow-production-git-sync
...
command: ["git-sync"]
args:
- "--repo=https://gitlab.discovery.wmnet:8443/repos/data-engineering/airflow-dags.git"
- "--root=/dags"
- "--link=airflow_dags"
- - "--ref=main"
+ - "--ref=main_airflow_3.3"
- "--period=300s"
- "--depth=1"
- "--sparse-checkout-file=/etc/gitsync/sparse-checkout.conf"
securityContext:
allowPrivilegeEscalation: false
...
airflow-test-k8s, airflow-hadoop-shell, Deployment (apps) has changed:
...
routed_via: production
component: hadoop-shell
role: toolbox
annotations:
checksum/secrets: 65333750984d5a3960b85d7927490a3baea64a4996174311b4a8fd2df39a39d0
- checksum/configuration: 0d8f274ee3fe859082a1b84082de50bef5deed076567598110ca4ef2b9c628ac
+ checksum/configuration: a5083eae5608f9124fefb280c36541330ffd0e6c330978b8210babd3efe76322
spec:
volumes:
- name: airflow-hadoop-configuration
configMap:
name: airflow-hadoop-configuration
...
- an-test-master1004.eqiad.wmnet
containers:
- name: "hadoop-shell"
command: ["sleep"]
args: ["infinity"]
- image: "docker-registry.discovery.wmnet/repos/data-engineering/airflow-dags:airflow-2.11.2-py3.11-2026-06-10-131616-6b0fbe1f9b58505d433c58641979ec063ddb15ee@sha256:18e5bf4bbd07aa0a8eb9a93d7f1dc52640f16d7bc973b3cca6398ea2c868702d"
+ image: "docker-registry.discovery.wmnet/repos/data-engineering/airflow-dags:airflow-3.3.1-py3.11-2026-09-23-160148-1ae06b2b0a30b6f058d913d298829f1b94caf266@sha256:2d70acfbc0c315327b5a57df4158debd28573d45dd03714341f9201b271d3064"
imagePullPolicy: IfNotPresent
env:
- name: HADOOP_CONF_DIR
value: /etc/hadoop/conf
- name: HIVE_CONF_DIR
...
airflow-test-k8s, airflow-kerberos, Deployment (apps) has changed:
...
release: production
routed_via: production
component: kerberos
annotations:
checksum/secrets: 65333750984d5a3960b85d7927490a3baea64a4996174311b4a8fd2df39a39d0
- checksum/configuration: 0d8f274ee3fe859082a1b84082de50bef5deed076567598110ca4ef2b9c628ac
+ checksum/configuration: a5083eae5608f9124fefb280c36541330ffd0e6c330978b8210babd3efe76322
spec:
securityContext:
fsGroup: 900
containers:
- name: "airflow-kerberos"
command: ["airflow"]
args:
- kerberos
- --pid
- /tmp/airflow-kerberos.pid
- image: "docker-registry.discovery.wmnet/repos/data-engineering/airflow-dags:airflow-2.11.2-py3.11-2026-06-10-131616-6b0fbe1f9b58505d433c58641979ec063ddb15ee@sha256:18e5bf4bbd07aa0a8eb9a93d7f1dc52640f16d7bc973b3cca6398ea2c868702d"
+ image: "docker-registry.discovery.wmnet/repos/data-engineering/airflow-dags:airflow-3.3.1-py3.11-2026-09-23-160148-1ae06b2b0a30b6f058d913d298829f1b94caf266@sha256:2d70acfbc0c315327b5a57df4158debd28573d45dd03714341f9201b271d3064"
imagePullPolicy: IfNotPresent
env:
- name: SERVICE_IDENTIFIER
value: airflow-production
- name: AIRFLOW_HOME
...
value: "False"
- name: ENVOY_SERVICE_NAME
value: airflow-envoy
- name: AIRFLOW_API_URL
value: "https://airflow-production-tls-service.airflow-test-k8s.svc.cluster.local:1025"
+ - name: AIRFLOW_SERVICE_TOKEN
+ value: "/tmp/airflow_service_token/token"
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop:
...
mountPath: /opt/airflow/secrets
- name: airflow-dags
readOnly: true
mountPath: /opt/airflow/dags
- name: airflow-kubernetes-pod-templates
- mountPath: /opt/airflow/pod_templates
+ mountPath: /opt/airflow/pod_templates
+ - name: airflow-service-token
+ mountPath: /tmp/airflow_service_token
- name: airflow-kerberos-client-config
mountPath: /etc/krb5.conf
subPath: krb5.conf
- name: airflow-kerberos-token
mountPath: /tmp/airflow_krb5_ccache
...
name: airflow-kubernetes-pod-templates
- name: airflow-logs
emptyDir: {}
- name: airflow-dags
persistentVolumeClaim:
- claimName: airflow-dags-pvc
+ claimName: airflow-dags-pvc
+ - name: airflow-service-token
+ projected:
+ sources:
+ - serviceAccountToken:
+ audience: airflow
+ expirationSeconds: 600
+ path: token
- name: airflow-kerberos-client-config
configMap:
name: airflow-kerberos-client-config
- name: airflow-kerberos-token
persistentVolumeClaim:
...
airflow-test-k8s, airflow-kubernetes-pod-templates, ConfigMap (v1) has changed:
...
- name: airflow-logs
emptyDir: {}
- name: airflow-dags
persistentVolumeClaim:
claimName: airflow-dags-pvc
+ - name: airflow-service-token
+ projected:
+ sources:
+ - serviceAccountToken:
+ audience: airflow
+ expirationSeconds: 600
+ path: token
- name: airflow-hadoop-configuration
configMap:
name: airflow-hadoop-configuration
- name: airflow-spark-configuration
configMap:
...
secret:
secretName: airflow-kerberos-keytab
priorityClassName: low-priority-pod
containers:
- name: base
- image: "docker-registry.discovery.wmnet/repos/data-engineering/airflow-dags:airflow-2.11.2-py3.11-2026-06-10-131616-6b0fbe1f9b58505d433c58641979ec063ddb15ee@sha256:18e5bf4bbd07aa0a8eb9a93d7f1dc52640f16d7bc973b3cca6398ea2c868702d"
+ image: "docker-registry.discovery.wmnet/repos/data-engineering/airflow-dags:airflow-3.3.1-py3.11-2026-09-23-160148-1ae06b2b0a30b6f058d913d298829f1b94caf266@sha256:2d70acfbc0c315327b5a57df4158debd28573d45dd03714341f9201b271d3064"
imagePullPolicy: IfNotPresent
env:
- name: SERVICE_IDENTIFIER
value: airflow-production
- name: AIRFLOW_HOME
...
- name: SCARF_ANALYTICS
value: "False"
- name: ENVOY_SERVICE_NAME
value: airflow-envoy
- name: AIRFLOW_API_URL
- value: "https://airflow-production-tls-service.airflow-test-k8s.svc.cluster.local:1025"
+ value: "https://airflow-production-tls-service.airflow-test-k8s.svc.cluster.local:1025"
+ - name: AIRFLOW_SERVICE_TOKEN
+ value: "/tmp/airflow_service_token/token"
- name: HADOOP_CONF_DIR
value: /etc/hadoop/conf
- name: HIVE_CONF_DIR
value: /etc/hadoop/conf
- name: ARROW_LIBHDFS_DIR
...
- name: airflow-dags
readOnly: true
mountPath: /opt/airflow/dags
- name: airflow-kubernetes-pod-templates
mountPath: /opt/airflow/pod_templates
+ - name: airflow-service-token
+ mountPath: /tmp/airflow_service_token
- name: airflow-hadoop-configuration
mountPath: /etc/hadoop/conf
- name: airflow-spark-configuration
mountPath: /etc/spark3/conf
- name: airflow-kerberos-client-config
...
- name: airflow-logs
emptyDir: {}
- name: airflow-dags
persistentVolumeClaim:
claimName: airflow-dags-pvc
+ - name: airflow-service-token
+ projected:
+ sources:
+ - serviceAccountToken:
+ audience: airflow
+ expirationSeconds: 600
+ path: token
- name: airflow-hadoop-configuration
configMap:
name: airflow-hadoop-configuration
- name: airflow-spark-configuration
configMap:
...
secret:
secretName: airflow-kerberos-keytab
priorityClassName: low-priority-pod
containers:
- name: base
- image: "docker-registry.discovery.wmnet/repos/data-engineering/airflow-dags:airflow-2.11.2-py3.11-2026-06-10-131616-6b0fbe1f9b58505d433c58641979ec063ddb15ee@sha256:18e5bf4bbd07aa0a8eb9a93d7f1dc52640f16d7bc973b3cca6398ea2c868702d"
+ image: "docker-registry.discovery.wmnet/repos/data-engineering/airflow-dags:airflow-3.3.1-py3.11-2026-09-23-160148-1ae06b2b0a30b6f058d913d298829f1b94caf266@sha256:2d70acfbc0c315327b5a57df4158debd28573d45dd03714341f9201b271d3064"
imagePullPolicy: IfNotPresent
env:
- name: SERVICE_IDENTIFIER
value: airflow-production
- name: AIRFLOW_HOME
...
- name: SCARF_ANALYTICS
value: "False"
- name: ENVOY_SERVICE_NAME
value: airflow-envoy
- name: AIRFLOW_API_URL
- value: "https://airflow-production-tls-service.airflow-test-k8s.svc.cluster.local:1025"
+ value: "https://airflow-production-tls-service.airflow-test-k8s.svc.cluster.local:1025"
+ - name: AIRFLOW_SERVICE_TOKEN
+ value: "/tmp/airflow_service_token/token"
- name: HADOOP_CONF_DIR
value: /etc/hadoop/conf
- name: HIVE_CONF_DIR
value: /etc/hadoop/conf
- name: ARROW_LIBHDFS_DIR
...
- name: airflow-dags
readOnly: true
mountPath: /opt/airflow/dags
- name: airflow-kubernetes-pod-templates
mountPath: /opt/airflow/pod_templates
+ - name: airflow-service-token
+ mountPath: /tmp/airflow_service_token
- name: airflow-hadoop-configuration
mountPath: /etc/hadoop/conf
- name: airflow-spark-configuration
mountPath: /etc/spark3/conf
- name: airflow-kerberos-client-config
...
- name: airflow-logs
emptyDir: {}
- name: airflow-dags
persistentVolumeClaim:
claimName: airflow-dags-pvc
+ - name: airflow-service-token
+ projected:
+ sources:
+ - serviceAccountToken:
+ audience: airflow
+ expirationSeconds: 600
+ path: token
- name: airflow-hadoop-configuration
configMap:
name: airflow-hadoop-configuration
- name: airflow-spark-configuration
configMap:
...
hostPath:
path: /usr/share/GeoIP
priorityClassName: low-priority-pod
containers:
- name: base
- image: "docker-registry.discovery.wmnet/repos/data-engineering/airflow-dags:airflow-2.11.2-py3.11-2026-06-10-131616-6b0fbe1f9b58505d433c58641979ec063ddb15ee@sha256:18e5bf4bbd07aa0a8eb9a93d7f1dc52640f16d7bc973b3cca6398ea2c868702d"
+ image: "docker-registry.discovery.wmnet/repos/data-engineering/airflow-dags:airflow-3.3.1-py3.11-2026-09-23-160148-1ae06b2b0a30b6f058d913d298829f1b94caf266@sha256:2d70acfbc0c315327b5a57df4158debd28573d45dd03714341f9201b271d3064"
imagePullPolicy: IfNotPresent
env:
- name: SERVICE_IDENTIFIER
value: airflow-production
- name: AIRFLOW_HOME
...
- name: SCARF_ANALYTICS
value: "False"
- name: ENVOY_SERVICE_NAME
value: airflow-envoy
- name: AIRFLOW_API_URL
- value: "https://airflow-production-tls-service.airflow-test-k8s.svc.cluster.local:1025"
+ value: "https://airflow-production-tls-service.airflow-test-k8s.svc.cluster.local:1025"
+ - name: AIRFLOW_SERVICE_TOKEN
+ value: "/tmp/airflow_service_token/token"
- name: HADOOP_CONF_DIR
value: /etc/hadoop/conf
- name: HIVE_CONF_DIR
value: /etc/hadoop/conf
- name: ARROW_LIBHDFS_DIR
...
- name: airflow-dags
readOnly: true
mountPath: /opt/airflow/dags
- name: airflow-kubernetes-pod-templates
mountPath: /opt/airflow/pod_templates
+ - name: airflow-service-token
+ mountPath: /tmp/airflow_service_token
- name: airflow-hadoop-configuration
mountPath: /etc/hadoop/conf
- name: airflow-spark-configuration
mountPath: /etc/spark3/conf
- name: airflow-kerberos-client-config
...
spec:
restartPolicy: Never
priorityClassName: low-priority-pod
containers:
- name: base
- image: "docker-registry.discovery.wmnet/repos/data-engineering/airflow-dags:airflow-2.11.2-py3.11-2026-06-10-131616-6b0fbe1f9b58505d433c58641979ec063ddb15ee@sha256:18e5bf4bbd07aa0a8eb9a93d7f1dc52640f16d7bc973b3cca6398ea2c868702d"
+ image: "docker-registry.discovery.wmnet/repos/data-engineering/airflow-dags:airflow-3.3.1-py3.11-2026-09-23-160148-1ae06b2b0a30b6f058d913d298829f1b94caf266@sha256:2d70acfbc0c315327b5a57df4158debd28573d45dd03714341f9201b271d3064"
imagePullPolicy: IfNotPresent
env:
- name: REQUESTS_CA_BUNDLE
value: /etc/ssl/certs/ca-certificates.crt
- name: AWS_REQUEST_CHECKSUM_CALCULATION
...
persistentVolumeClaim:
claimName: airflow-kerberos-token-pvc
priorityClassName: low-priority-pod
containers:
- name: base
- image: "docker-registry.discovery.wmnet/repos/data-engineering/airflow-dags:airflow-2.11.2-py3.11-2026-06-10-131616-6b0fbe1f9b58505d433c58641979ec063ddb15ee@sha256:18e5bf4bbd07aa0a8eb9a93d7f1dc52640f16d7bc973b3cca6398ea2c868702d"
+ image: "docker-registry.discovery.wmnet/repos/data-engineering/airflow-dags:airflow-3.3.1-py3.11-2026-09-23-160148-1ae06b2b0a30b6f058d913d298829f1b94caf266@sha256:2d70acfbc0c315327b5a57df4158debd28573d45dd03714341f9201b271d3064"
imagePullPolicy: IfNotPresent
env:
- name: HADOOP_CONF_DIR
value: /etc/hadoop/conf
- name: HIVE_CONF_DIR
...
airflow-test-k8s, airflow-scheduler, Deployment (apps) has changed:
...
release: production
routed_via: production
component: scheduler
annotations:
checksum/secrets: 65333750984d5a3960b85d7927490a3baea64a4996174311b4a8fd2df39a39d0
- checksum/configuration: 0d8f274ee3fe859082a1b84082de50bef5deed076567598110ca4ef2b9c628ac
+ checksum/configuration: a5083eae5608f9124fefb280c36541330ffd0e6c330978b8210babd3efe76322
checksum/tls-config: e7022733e316206a4aa5d56a48d4bb1574af43edc1f86be5d18704c479b1aedd
envoyproxy.io/scrape: "true"
envoyproxy.io/port: "1667"
spec:
serviceAccountName: airflow
+ initContainers:
+ - name: airflow-production-wait-for-initdb
+ image: "docker-registry.discovery.wmnet/repos/data-engineering/airflow-dags:airflow-3.3.1-py3.11-2026-09-23-160148-1ae06b2b0a30b6f058d913d298829f1b94caf266@sha256:2d70acfbc0c315327b5a57df4158debd28573d45dd03714341f9201b271d3064"
+ imagePullPolicy: IfNotPresent
+ command: ["airflow"]
+ args:
+ - db
+ - check-migrations
+ - --migration-wait-timeout=60
+ env:
+ - name: SERVICE_IDENTIFIER
+ value: airflow-production
+ - name: AIRFLOW_HOME
+ value: "/opt/airflow"
+ - name: AIRFLOW__API_AUTH__JWT_SECRET
+ valueFrom:
+ secretKeyRef:
+ name: airflow-production-secret-config
+ key: airflow__api_auth__jwt_secret
+ - name: AIRFLOW__CORE__FERNET_KEY
+ valueFrom:
+ secretKeyRef:
+ name: airflow-production-secret-config
+ key: airflow__core__fernet_key
+ - name: AIRFLOW__WEBSERVER__SECRET_KEY
+ valueFrom:
+ secretKeyRef:
+ name: airflow-production-secret-config
+ key: airflow__webserver__secret_key
+ - name: PG_HOST
+ valueFrom:
+ secretKeyRef:
+ name: postgresql-airflow-test-k8s-app
+ key: host
+ - name: PG_URI
+ valueFrom:
+ secretKeyRef:
+ name: postgresql-airflow-test-k8s-app
+ key: uri
+ - name: POOLER_NAME
+ value: postgresql-airflow-test-k8s-pooler-rw
+ - name: AIRFLOW__DATABASE__SQL_ALCHEMY_POOL_ENABLED
+ value: "False"
+ - name: AIRFLOW_INSTANCE_NAME
+ value: test-k8s
+ - name: AIRFLOW_ENVIRONMENT_NAME
+ value: wmf
+ - name: AIRFLOW_SCHEDULER_HOSTNAME
+ value: airflow-scheduler
+ - name: KRB5CCNAME
+ value: /tmp/airflow_krb5_ccache/krb5cc
+ - name: KRB5_CONFIG
+ value: /etc/krb5.conf
+ - name: AIRFLOW_APPOWNER
+ value: analytics
+ - name: AIRFLOW_KERBEROS_HOSTNAME
+ value: airflow-test-k8s.discovery.wmnet
+ - name: KRB5_PRINCIPAL
+ value: analytics/airflow-test-k8s.discovery.wmnet
+ - name: REQUESTS_CA_BUNDLE
+ value: /etc/ssl/certs/ca-certificates.crt
+ - name: AWS_REQUEST_CHECKSUM_CALCULATION
+ value: WHEN_REQUIRED
+ - name: AWS_RESPONSE_CHECKSUM_VALIDATION
+ value: WHEN_REQUIRED
+ - name: SCARF_ANALYTICS
+ value: "False"
+ - name: ENVOY_SERVICE_NAME
+ value: airflow-envoy
+ - name: AIRFLOW_API_URL
+ value: "https://airflow-production-tls-service.airflow-test-k8s.svc.cluster.local:1025"
+ - name: AIRFLOW_SERVICE_TOKEN
+ value: "/tmp/airflow_service_token/token"
+ resources:
+ requests:
+ cpu: 1000m
+ memory: 2Gi
+ limits:
+ cpu: 1000m
+ memory: 2Gi
+ securityContext:
+ allowPrivilegeEscalation: false
+ capabilities:
+ drop:
+ - ALL
+ runAsNonRoot: true
+ seccompProfile:
+ type: RuntimeDefault
+ volumeMounts:
+ - name: airflow-config
+ mountPath: /opt/airflow/airflow.cfg
+ subPath: airflow.cfg
+ - name: airflow-webserver-config
+ mountPath: /opt/airflow/auth_manager.py
+ subPath: auth_manager.py
+ - name: airflow-webserver-config
+ mountPath: /opt/airflow/webserver_config.py
+ subPath: webserver_config.py
+ - name: airflow-webserver-config
+ mountPath: /opt/airflow/airflow_local_settings.py
+ subPath: airflow_local_settings.py
+ - name: airflow-import-config
+ mountPath: /opt/airflow/import_helper.py
+ subPath: import_helper.py
+ - name: airflow-import-config
+ mountPath: /opt/airflow/airflowfilter
+ subPath: airflowfilter
+ - name: airflow-logs
+ mountPath: /opt/airflow/logs
+ - name: airflow-bash-executables
+ mountPath: /opt/airflow/usr/bin
+ - name: airflow-connections-variables
+ mountPath: /opt/airflow/secrets
+ - name: airflow-dags
+ readOnly: true
+ mountPath: /opt/airflow/dags
+ - name: airflow-kubernetes-pod-templates
+ mountPath: /opt/airflow/pod_templates
+ - name: airflow-service-token
+ mountPath: /tmp/airflow_service_token
containers:
- name: airflow-production
- image: "docker-registry.discovery.wmnet/repos/data-engineering/airflow-dags:airflow-2.11.2-py3.11-2026-06-10-131616-6b0fbe1f9b58505d433c58641979ec063ddb15ee@sha256:18e5bf4bbd07aa0a8eb9a93d7f1dc52640f16d7bc973b3cca6398ea2c868702d"
+ image: "docker-registry.discovery.wmnet/repos/data-engineering/airflow-dags:airflow-3.3.1-py3.11-2026-09-23-160148-1ae06b2b0a30b6f058d913d298829f1b94caf266@sha256:2d70acfbc0c315327b5a57df4158debd28573d45dd03714341f9201b271d3064"
imagePullPolicy: IfNotPresent
command: ["airflow"]
args: ["scheduler", "--pid" , "/tmp/airflow-scheduler.pid"]
ports:
- containerPort: 8765
...
- name: SCARF_ANALYTICS
value: "False"
- name: ENVOY_SERVICE_NAME
value: airflow-envoy
- name: AIRFLOW_API_URL
- value: "https://airflow-production-tls-service.airflow-test-k8s.svc.cluster.local:1025"
+ value: "https://airflow-production-tls-service.airflow-test-k8s.svc.cluster.local:1025"
+ - name: AIRFLOW_SERVICE_TOKEN
+ value: "/tmp/airflow_service_token/token"
resources:
requests:
cpu: "1"
memory: 2Gi
limits:
...
mountPath: /opt/airflow/secrets
- name: airflow-dags
readOnly: true
mountPath: /opt/airflow/dags
- name: airflow-kubernetes-pod-templates
- mountPath: /opt/airflow/pod_templates
+ mountPath: /opt/airflow/pod_templates
+ - name: airflow-service-token
+ mountPath: /tmp/airflow_service_token
- name: airflow-kerberos-client-config
mountPath: /etc/krb5.conf
subPath: krb5.conf
- name: airflow-kerberos-token
mountPath: /tmp/airflow_krb5_ccache
...
name: airflow-kubernetes-pod-templates
- name: airflow-logs
emptyDir: {}
- name: airflow-dags
persistentVolumeClaim:
- claimName: airflow-dags-pvc
+ claimName: airflow-dags-pvc
+ - name: airflow-service-token
+ projected:
+ sources:
+ - serviceAccountToken:
+ audience: airflow
+ expirationSeconds: 600
+ path: token
- name: airflow-kerberos-client-config
configMap:
name: airflow-kerberos-client-config
- name: airflow-kerberos-token
persistentVolumeClaim:
...
airflow-test-k8s, airflow-task-shell, Deployment (apps) has changed:
...
routed_via: production
component: task-pod
role: toolbox
annotations:
checksum/secrets: 65333750984d5a3960b85d7927490a3baea64a4996174311b4a8fd2df39a39d0
- checksum/configuration: 0d8f274ee3fe859082a1b84082de50bef5deed076567598110ca4ef2b9c628ac
+ checksum/configuration: a5083eae5608f9124fefb280c36541330ffd0e6c330978b8210babd3efe76322
spec:
hostAliases:
- ip: 10.64.159.4
hostnames:
- an-test-master1003.eqiad.wmnet
...
defaultMode: 0777
containers:
- name: "task-shell"
command: ["sleep"]
args: ["infinity"]
- image: "docker-registry.discovery.wmnet/repos/data-engineering/airflow-dags:airflow-2.11.2-py3.11-2026-06-10-131616-6b0fbe1f9b58505d433c58641979ec063ddb15ee@sha256:18e5bf4bbd07aa0a8eb9a93d7f1dc52640f16d7bc973b3cca6398ea2c868702d"
+ image: "docker-registry.discovery.wmnet/repos/data-engineering/airflow-dags:airflow-3.3.1-py3.11-2026-09-23-160148-1ae06b2b0a30b6f058d913d298829f1b94caf266@sha256:2d70acfbc0c315327b5a57df4158debd28573d45dd03714341f9201b271d3064"
imagePullPolicy: IfNotPresent
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop:
...
airflow-test-k8s, airflow-webserver, Deployment (apps) has changed:
...
release: production
routed_via: production
component: webserver
annotations:
checksum/secrets: 65333750984d5a3960b85d7927490a3baea64a4996174311b4a8fd2df39a39d0
- checksum/configuration: 0d8f274ee3fe859082a1b84082de50bef5deed076567598110ca4ef2b9c628ac
+ checksum/configuration: a5083eae5608f9124fefb280c36541330ffd0e6c330978b8210babd3efe76322
checksum/tls-config: e7022733e316206a4aa5d56a48d4bb1574af43edc1f86be5d18704c479b1aedd
envoyproxy.io/scrape: "true"
envoyproxy.io/port: "1667"
spec:
serviceAccountName: airflow
initContainers:
- name: airflow-production-initdb
command: ["airflow"]
args: ["db", "migrate"]
- image: "docker-registry.discovery.wmnet/repos/data-engineering/airflow-dags:airflow-2.11.2-py3.11-2026-06-10-131616-6b0fbe1f9b58505d433c58641979ec063ddb15ee@sha256:18e5bf4bbd07aa0a8eb9a93d7f1dc52640f16d7bc973b3cca6398ea2c868702d"
+ image: "docker-registry.discovery.wmnet/repos/data-engineering/airflow-dags:airflow-3.3.1-py3.11-2026-09-23-160148-1ae06b2b0a30b6f058d913d298829f1b94caf266@sha256:2d70acfbc0c315327b5a57df4158debd28573d45dd03714341f9201b271d3064"
imagePullPolicy: IfNotPresent
env:
- name: SERVICE_IDENTIFIER
value: airflow-production
- name: AIRFLOW_HOME
...
value: "False"
- name: ENVOY_SERVICE_NAME
value: airflow-envoy
- name: AIRFLOW_API_URL
value: "https://airflow-production-tls-service.airflow-test-k8s.svc.cluster.local:1025"
+ - name: AIRFLOW_SERVICE_TOKEN
+ value: "/tmp/airflow_service_token/token"
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop:
...
- name: airflow-dags
readOnly: true
mountPath: /opt/airflow/dags
- name: airflow-kubernetes-pod-templates
mountPath: /opt/airflow/pod_templates
+ - name: airflow-service-token
+ mountPath: /tmp/airflow_service_token
containers:
# The main application container
- name: airflow-production
- image: "docker-registry.discovery.wmnet/repos/data-engineering/airflow-dags:airflow-2.11.2-py3.11-2026-06-10-131616-6b0fbe1f9b58505d433c58641979ec063ddb15ee@sha256:18e5bf4bbd07aa0a8eb9a93d7f1dc52640f16d7bc973b3cca6398ea2c868702d"
+ image: "docker-registry.discovery.wmnet/repos/data-engineering/airflow-dags:airflow-3.3.1-py3.11-2026-09-23-160148-1ae06b2b0a30b6f058d913d298829f1b94caf266@sha256:2d70acfbc0c315327b5a57df4158debd28573d45dd03714341f9201b271d3064"
imagePullPolicy: IfNotPresent
command:
- airflow
args:
- - "webserver"
+ - "api-server"
- "--pid"
- "/tmp/airflow-webserver.pid"
ports:
- containerPort: 8080
livenessProbe:
tcpSocket:
port: 8080
readinessProbe:
httpGet:
- path: /health
+ path: /api/v2/monitor/health
port: 8080
env:
- name: SERVICE_IDENTIFIER
value: airflow-production
- name: AIRFLOW_HOME
...
- name: SCARF_ANALYTICS
value: "False"
- name: ENVOY_SERVICE_NAME
value: airflow-envoy
- name: AIRFLOW_API_URL
- value: "https://airflow-production-tls-service.airflow-test-k8s.svc.cluster.local:1025"
+ value: "https://airflow-production-tls-service.airflow-test-k8s.svc.cluster.local:1025"
+ - name: AIRFLOW_SERVICE_TOKEN
+ value: "/tmp/airflow_service_token/token"
resources:
requests:
cpu: 1000m
memory: 2Gi
limits:
...
mountPath: /opt/airflow/secrets
- name: airflow-dags
readOnly: true
mountPath: /opt/airflow/dags
- name: airflow-kubernetes-pod-templates
- mountPath: /opt/airflow/pod_templates
+ mountPath: /opt/airflow/pod_templates
+ - name: airflow-service-token
+ mountPath: /tmp/airflow_service_token
- name: airflow-kerberos-client-config
mountPath: /etc/krb5.conf
subPath: krb5.conf
- name: airflow-kerberos-token
mountPath: /tmp/airflow_krb5_ccache
...
name: airflow-kubernetes-pod-templates
- name: airflow-logs
emptyDir: {}
- name: airflow-dags
persistentVolumeClaim:
- claimName: airflow-dags-pvc
+ claimName: airflow-dags-pvc
+ - name: airflow-service-token
+ projected:
+ sources:
+ - serviceAccountToken:
+ audience: airflow
+ expirationSeconds: 600
+ path: token
- name: airflow-kerberos-client-config
configMap:
name: airflow-kerberos-client-config
- name: airflow-kerberos-token
persistentVolumeClaim:
...
airflow-test-k8s, airflow-webserver-config, ConfigMap (v1) has changed:
...
"kubernetes.io:namespace:airflow-test-k8s": ["Admin"],
}
AUTH_K8S_TOKEN_VERIFICATION_OPTIONS = {
"audience": [
- "",
+ "airflow",
]
}
# Flask-WTF
WTF_CSRF_ENABLED = True
...
# Airflow 2.x
from airflow.www.utils import UIAlert as BaseUIAlert
def UIAlert(text, html, category):
return BaseUIAlert(text, html=html, category=category)
DASHBOARD_UIALERTS = [
+ UIAlert(
+ text="""<code>git-sync</code> is currently pulling the <code><a href="https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/tree/main_airflow_3.3">main_airflow_3.3</a></code> ref from <code>airflow-dags</code>.""",
+ html=True,
+ category="warning",
+ ),
]
try:
# Airflow 3.3+
from airflow.providers.cncf.kubernetes.utils.xcom_sidecar import PodDefaults
except ImportError:
...
airflow-test-k8s, airflow-dag-processor, Deployment (apps) has been added:
-
+ # Source: airflow/templates/deployment.yaml
+ apiVersion: apps/v1
+ kind: Deployment
+ metadata:
+ name: airflow-dag-processor
+ labels:
+ app: airflow
+ chart: airflow-0.0.182
+ release: production
+ heritage: Helm
+ component: dag-processor
+ spec:
+ selector:
+ matchLabels:
+ app: airflow
+ release: production
+ replicas: 1
+ template:
+ metadata:
+ labels:
+ app: airflow
+ release: production
+ routed_via: production
+ component: dag-processor
+ annotations:
+ checksum/secrets: 65333750984d5a3960b85d7927490a3baea64a4996174311b4a8fd2df39a39d0
+ checksum/configuration: a5083eae5608f9124fefb280c36541330ffd0e6c330978b8210babd3efe76322
+ checksum/tls-config: e7022733e316206a4aa5d56a48d4bb1574af43edc1f86be5d18704c479b1aedd
+ envoyproxy.io/scrape: "true"
+ envoyproxy.io/port: "1667"
+ spec:
+ serviceAccountName: airflow
+ initContainers:
+ - name: airflow-production-wait-for-initdb
+ image: "docker-registry.discovery.wmnet/repos/data-engineering/airflow-dags:airflow-3.3.1-py3.11-2026-09-23-160148-1ae06b2b0a30b6f058d913d298829f1b94caf266@sha256:2d70acfbc0c315327b5a57df4158debd28573d45dd03714341f9201b271d3064"
+ imagePullPolicy: IfNotPresent
+ command: ["airflow"]
+ args:
+ - db
+ - check-migrations
+ - --migration-wait-timeout=60
+ env:
+ - name: SERVICE_IDENTIFIER
+ value: airflow-production
+ - name: AIRFLOW_HOME
+ value: "/opt/airflow"
+ - name: AIRFLOW__API_AUTH__JWT_SECRET
+ valueFrom:
+ secretKeyRef:
+ name: airflow-production-secret-config
+ key: airflow__api_auth__jwt_secret
+ - name: AIRFLOW__CORE__FERNET_KEY
+ valueFrom:
+ secretKeyRef:
+ name: airflow-production-secret-config
+ key: airflow__core__fernet_key
+ - name: AIRFLOW__WEBSERVER__SECRET_KEY
+ valueFrom:
+ secretKeyRef:
+ name: airflow-production-secret-config
+ key: airflow__webserver__secret_key
+ - name: PG_HOST
+ valueFrom:
+ secretKeyRef:
+ name: postgresql-airflow-test-k8s-app
+ key: host
+ - name: PG_URI
+ valueFrom:
+ secretKeyRef:
+ name: postgresql-airflow-test-k8s-app
+ key: uri
+ - name: POOLER_NAME
+ value: postgresql-airflow-test-k8s-pooler-rw
+ - name: AIRFLOW__DATABASE__SQL_ALCHEMY_POOL_ENABLED
+ value: "False"
+ - name: AIRFLOW_INSTANCE_NAME
+ value: test-k8s
+ - name: AIRFLOW_ENVIRONMENT_NAME
+ value: wmf
+ - name: AIRFLOW_SCHEDULER_HOSTNAME
+ value: airflow-scheduler
+ - name: KRB5CCNAME
+ value: /tmp/airflow_krb5_ccache/krb5cc
+ - name: KRB5_CONFIG
+ value: /etc/krb5.conf
+ - name: AIRFLOW_APPOWNER
+ value: analytics
+ - name: AIRFLOW_KERBEROS_HOSTNAME
+ value: airflow-test-k8s.discovery.wmnet
+ - name: KRB5_PRINCIPAL
+ value: analytics/airflow-test-k8s.discovery.wmnet
+ - name: REQUESTS_CA_BUNDLE
+ value: /etc/ssl/certs/ca-certificates.crt
+ - name: AWS_REQUEST_CHECKSUM_CALCULATION
+ value: WHEN_REQUIRED
+ - name: AWS_RESPONSE_CHECKSUM_VALIDATION
+ value: WHEN_REQUIRED
+ - name: SCARF_ANALYTICS
+ value: "False"
+ - name: ENVOY_SERVICE_NAME
+ value: airflow-envoy
+ - name: AIRFLOW_API_URL
+ value: "https://airflow-production-tls-service.airflow-test-k8s.svc.cluster.local:1025"
+ - name: AIRFLOW_SERVICE_TOKEN
+ value: "/tmp/airflow_service_token/token"
+ resources:
+ requests:
+ cpu: 1000m
+ memory: 2Gi
+ limits:
+ cpu: 1000m
+ memory: 2Gi
+ securityContext:
+ allowPrivilegeEscalation: false
+ capabilities:
+ drop:
+ - ALL
+ runAsNonRoot: true
+ seccompProfile:
+ type: RuntimeDefault
+ volumeMounts:
+ - name: airflow-config
+ mountPath: /opt/airflow/airflow.cfg
+ subPath: airflow.cfg
+ - name: airflow-webserver-config
+ mountPath: /opt/airflow/auth_manager.py
+ subPath: auth_manager.py
+ - name: airflow-webserver-config
+ mountPath: /opt/airflow/webserver_config.py
+ subPath: webserver_config.py
+ - name: airflow-webserver-config
+ mountPath: /opt/airflow/airflow_local_settings.py
+ subPath: airflow_local_settings.py
+ - name: airflow-import-config
+ mountPath: /opt/airflow/import_helper.py
+ subPath: import_helper.py
+ - name: airflow-import-config
+ mountPath: /opt/airflow/airflowfilter
+ subPath: airflowfilter
+ - name: airflow-logs
+ mountPath: /opt/airflow/logs
+ - name: airflow-bash-executables
+ mountPath: /opt/airflow/usr/bin
+ - name: airflow-connections-variables
+ mountPath: /opt/airflow/secrets
+ - name: airflow-dags
+ readOnly: true
+ mountPath: /opt/airflow/dags
+ - name: airflow-kubernetes-pod-templates
+ mountPath: /opt/airflow/pod_templates
+ - name: airflow-service-token
+ mountPath: /tmp/airflow_service_token
+ containers:
+ - name: airflow-production
+ image: "docker-registry.discovery.wmnet/repos/data-engineering/airflow-dags:airflow-3.3.1-py3.11-2026-09-23-160148-1ae06b2b0a30b6f058d913d298829f1b94caf266@sha256:2d70acfbc0c315327b5a57df4158debd28573d45dd03714341f9201b271d3064"
+ imagePullPolicy: IfNotPresent
+ command: ["airflow"]
+ args: ["dag-processor", "--pid" , "/tmp/airflow-dag-processor.pid"]
+ livenessProbe:
+ exec:
+ command:
+ - sh
+ - -c
+ - CONNECTION_CHECK_MAX_COUNT=0 AIRFLOW__LOGGING__LOGGING_LEVEL=ERROR airflow jobs
+ check --local --job-type DagProcessorJob
+ failureThreshold: 5
+ initialDelaySeconds: 10
+ periodSeconds: 60
+ timeoutSeconds: 20
+ env:
+ - name: SERVICE_IDENTIFIER
+ value: airflow-production
+ - name: AIRFLOW_HOME
+ value: "/opt/airflow"
+ - name: AIRFLOW__API_AUTH__JWT_SECRET
+ valueFrom:
+ secretKeyRef:
+ name: airflow-production-secret-config
+ key: airflow__api_auth__jwt_secret
+ - name: AIRFLOW__CORE__FERNET_KEY
+ valueFrom:
+ secretKeyRef:
+ name: airflow-production-secret-config
+ key: airflow__core__fernet_key
+ - name: AIRFLOW__WEBSERVER__SECRET_KEY
+ valueFrom:
+ secretKeyRef:
+ name: airflow-production-secret-config
+ key: airflow__webserver__secret_key
+ - name: PG_HOST
+ valueFrom:
+ secretKeyRef:
+ name: postgresql-airflow-test-k8s-app
+ key: host
+ - name: PG_URI
+ valueFrom:
+ secretKeyRef:
+ name: postgresql-airflow-test-k8s-app
+ key: uri
+ - name: POOLER_NAME
+ value: postgresql-airflow-test-k8s-pooler-rw
+ - name: AIRFLOW__DATABASE__SQL_ALCHEMY_POOL_ENABLED
+ value: "False"
+ - name: AIRFLOW_INSTANCE_NAME
+ value: test-k8s
+ - name: AIRFLOW_ENVIRONMENT_NAME
+ value: wmf
+ - name: AIRFLOW_SCHEDULER_HOSTNAME
+ value: airflow-scheduler
+ - name: KRB5CCNAME
+ value: /tmp/airflow_krb5_ccache/krb5cc
+ - name: KRB5_CONFIG
+ value: /etc/krb5.conf
+ - name: AIRFLOW_APPOWNER
+ value: analytics
+ - name: AIRFLOW_KERBEROS_HOSTNAME
+ value: airflow-test-k8s.discovery.wmnet
+ - name: KRB5_PRINCIPAL
+ value: analytics/airflow-test-k8s.discovery.wmnet
+ - name: REQUESTS_CA_BUNDLE
+ value: /etc/ssl/certs/ca-certificates.crt
+ - name: AWS_REQUEST_CHECKSUM_CALCULATION
+ value: WHEN_REQUIRED
+ - name: AWS_RESPONSE_CHECKSUM_VALIDATION
+ value: WHEN_REQUIRED
+ - name: SCARF_ANALYTICS
+ value: "False"
+ - name: ENVOY_SERVICE_NAME
+ value: airflow-envoy
+ - name: AIRFLOW_API_URL
+ value: "https://airflow-production-tls-service.airflow-test-k8s.svc.cluster.local:1025"
+ - name: AIRFLOW_SERVICE_TOKEN
+ value: "/tmp/airflow_service_token/token"
+ resources:
+ requests:
+ cpu: "0.5"
+ memory: 1Gi
+ limits:
+ cpu: "0.5"
+ memory: 2Gi
+ securityContext:
+ allowPrivilegeEscalation: false
+ capabilities:
+ drop:
+ - ALL
+ runAsNonRoot: true
+ seccompProfile:
+ type: RuntimeDefault
+ volumeMounts:
+ - name: airflow-config
+ mountPath: /opt/airflow/airflow.cfg
+ subPath: airflow.cfg
+ - name: airflow-webserver-config
+ mountPath: /opt/airflow/auth_manager.py
+ subPath: auth_manager.py
+ - name: airflow-webserver-config
+ mountPath: /opt/airflow/webserver_config.py
+ subPath: webserver_config.py
+ - name: airflow-webserver-config
+ mountPath: /opt/airflow/airflow_local_settings.py
+ subPath: airflow_local_settings.py
+ - name: airflow-import-config
+ mountPath: /opt/airflow/import_helper.py
+ subPath: import_helper.py
+ - name: airflow-import-config
+ mountPath: /opt/airflow/airflowfilter
+ subPath: airflowfilter
+ - name: airflow-logs
+ mountPath: /opt/airflow/logs
+ - name: airflow-bash-executables
+ mountPath: /opt/airflow/usr/bin
+ - name: airflow-connections-variables
+ mountPath: /opt/airflow/secrets
+ - name: airflow-dags
+ readOnly: true
+ mountPath: /opt/airflow/dags
+ - name: airflow-kubernetes-pod-templates
+ mountPath: /opt/airflow/pod_templates
+ - name: airflow-service-token
+ mountPath: /tmp/airflow_service_token
+ volumes:
+ - name: airflow-config
+ configMap:
+ name: airflow-config
+ - name: airflow-webserver-config
+ configMap:
+ name: airflow-webserver-config
+ - name: airflow-import-config
+ configMap:
+ name: airflow-import-config
+ - name: airflow-bash-executables
+ configMap:
+ name: airflow-bash-executables
+ defaultMode: 0777
+ - name: airflow-connections-variables
+ secret:
+ secretName: airflow-connections-variables
+ - name: airflow-kubernetes-pod-templates
+ configMap:
+ name: airflow-kubernetes-pod-templates
+ - name: airflow-logs
+ emptyDir: {}
+ - name: airflow-dags
+ persistentVolumeClaim:
+ claimName: airflow-dags-pvc
+ - name: airflow-service-token
+ projected:
+ sources:
+ - serviceAccountToken:
+ audience: airflow
+ expirationSeconds: 600
+ path: token
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits