We have received multiple alerts of this nature:
AirflowDeploymentUnavailable data-platform (airflow-gitsync airflow-platform-eng k8s-dse critical eqiad prometheus)
Upon investigation, we can confirm that the airflow-gitsync pods are in a crashloop state.
btullis@deploy1003:~$ kube-env airflow-platform-eng-deploy dse-k8s-eqiad btullis@deploy1003:~$ kubectl get pods NAME READY STATUS RESTARTS AGE airflow-envoy-696fdc57fc-g55cl 1/1 Running 0 22d airflow-gitsync-54845464d5-qpt8n 0/1 CrashLoopBackOff 13 (105s ago) 22d airflow-hadoop-shell-56c9455c87-f94mr 1/1 Running 0 22d airflow-kerberos-5b48d6d54f-7w44s 1/1 Running 0 22d airflow-scheduler-7984fb654c-246z7 1/1 Running 1 (11d ago) 12d airflow-statsd-85fc54bb4-9q847 1/1 Running 0 34d airflow-task-shell-64b8ff584-ff49j 1/1 Running 0 22d airflow-webserver-7bf6b5c7d8-qhbd2 2/2 Running 0 22d postgresql-airflow-platform-eng-1 1/1 Running 0 34d postgresql-airflow-platform-eng-2 1/1 Running 0 34d postgresql-airflow-platform-eng-pooler-rw-6f76b54fb6-lzsfp 1/1 Running 0 34d postgresql-airflow-platform-eng-pooler-rw-6f76b54fb6-nvskj 1/1 Running 0 34d postgresql-airflow-platform-eng-pooler-rw-6f76b54fb6-spvs5 1/1 Running 0 34d <snip>
I checked the logs of the pod and it said this.
btullis@deploy1003:~$ kubectl logs airflow-gitsync-54845464d5-qpt8n
INFO: detected pid 1, running init handler
{"logger":"","ts":"2026-08-19 08:48:03.904420","caller":{"file":"main.go","line":568},"level":0,"msg":"starting up","version":"4.2.4","pid":11,"uid":900,"gid":900,"home":"/home/runuser","flags":["--depth=1","--link=airflow_dags","--period=5m0s","--ref=main","--repo=https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags.git","--root=/dags","--sparse-checkout-file=/etc/gitsync/sparse-checkout.conf"]}
{"logger":"","ts":"2026-08-19 08:48:03.905684","caller":{"file":"main.go","line":673},"level":0,"msg":"git version","version":"git version 2.39.2"}
{"logger":"","ts":"2026-08-19 08:50:18.748648","caller":{"file":"main.go","line":855},"msg":"too many failures, aborting","error":"Run(git fetch https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags.git main --verbose --no-progress --prune --no-auto-gc --depth 1): context deadline exceeded: { stdout: \"\", stderr: \"fatal: unable to access 'https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags.git/': Failed to connect to gitlab.wikimedia.org port 443 after 134536 ms: Couldn't connect to server\" }","failCount":1}I deleted the pod and watched the recovery,
btullis@deploy1003:~$ kubectl delete pod airflow-gitsync-54845464d5-qpt8n pod "airflow-gitsync-54845464d5-qpt8n" deleted
But pretty soon it started going into a crashloop again.
btullis@deploy1003:~$ kubectl get pods NAME READY STATUS RESTARTS AGE airflow-envoy-696fdc57fc-g55cl 1/1 Running 0 22d airflow-gitsync-54845464d5-b5nr4 0/1 CrashLoopBackOff 2 (10s ago) 7m12s `