User Details
- User Since
- Mar 30 2026, 1:14 PM (31 w, 6 d)
- Availability
- Available
- IRC Nick
- atsukoito
- LDAP User
- Atsuko
- MediaWiki User
- Atsuko-WMF [ Global Accounts ]
Fri, Nov 6
This happens because we have a resource with non-templated name (in this case, PersistentVolumeClaim/dev-metrics-git-pipeline), defined in helmfile.d/dse-k8s-services/airflow-main/values-production.yaml,
kubernetes_pod_operator:
persistent_volume_claims:
# CephFS disk for the dev-metrics git commit pipeline.
# Treated as a rebuildable cache of the collected repo clones.
# Mounted by the DAG.
- claim_name: dev-metrics-git-pipeline
size: 20Gi
storage_class: ceph-cephfs-ssdThere was rolling restart of K8s nodes around the time operator pod has been reported missing,
2026-11-04, 16:52:46 UTC 1 heartbeat timeout Task did not emit heartbeat within time limit (300 seconds) and will be terminated. See https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/tasks.html#zombie-undead-tasks
We have upgraded 6 instances and was ready to upgrade 7th (ml), but we are now held back with the investigation of performance problems T443252: airflow-dumps webserver performance, as well, and, also, whole SRE DPE are on SRE summit next week. We'll re-group and announce our next course of actions next week.
Thu, Nov 5
We hit the performance issues with airflow-dumps, I think we should take another look on the DAG structure for ML team to check if it is similar to the ones causing performance issues.
Wed, Nov 4
dumps UI operation is recovered
We now found that there are a few more hot URLs
- /ui/structure/structure_data?dag_id=mediawiki_dumps_sql_xml_large_a_to_z_full&external_dependencies=false&version_number=2997
- /ui/structure/structure_data?dag_id=mediawiki_dumps_sql_xml_large_a_to_z_full
- cover report P97100
- pyinstrument chart
- commit in question that introduces the slow code is b1d87bbefaa1216893f48fd48341f558c79ee53a
Tue, Nov 3
Collecting flamegraphs with pyinstrument
Mon, Nov 2
And I see a few recursions here on a smaller DAG https://phabricator.wikimedia.org/P97091
Tracing in-place, running in webserver container, with the token from the UI
python3 -m trace --count --file /tmp/report -C /tmp/cover /home/app/.local/bin/airflow api-server --port 8088 -l /tmp/log -w 1 --apps core &
time python3 -c "import requests; print(requests.get('http://localhost:8088/ui/grid/structure/mediawiki_dumps_sql_xml_large_a_to_z_full?include_upstream=false&include_downstream=false&limit=1&order_by=-run_after', headers={'Authorization': 'Bearer <token>'}).text)"it seems that https://github.com/apache/airflow/issues/57776 is what we see
One of endpoints times out and takes about 4 minutes
runuser@airflow-webserver-98c9664b-lv7ml:/opt/airflow$ time python3 -c "import requests; requests.get('http://localhost:8080/ui/grid/structure/mediawiki_dumps_sql_xml_large_a_to_z_full?include_upstream=false&include_downstream=false&limit=1&order_by=-run_after', headers={'Authorization': 'Bearer ...'}).text"Note:
After changing volume group in topolvm
- I needed to restart lvmd with systemctl kill --signal=9 lvmd, otherwise it was stuck
- Annotations weren't removed from nodes, I needed to remove those manually and restart topolvm-node
Seems that topolvm didn't update annotations on the host even after restarting everything
kubectl --namespace=kube-system get node dse-k8s-worker1001.eqiad.wmnet -o yaml grep topolvm
capacity.topolvm.io/00default: "1989102534656"
capacity.topolvm.io/balanced: "1989102534656"
capacity.topolvm.io/capacity: "1989102534656"
csi.volume.kubernetes.io/nodeid: '{"cephfs.csi.ceph.com":"dse-k8s-worker1001.eqiad.wmnet","rbd.csi.ceph.com":"dse-k8s-worker1001.eqiad.wmnet","topolvm.io":"dse-k8s-worker1001.eqiad.wmnet"}'
topology.topolvm.io/node: dse-k8s-worker1001.eqiad.wmnetFri, Oct 30
I feel like WDQS should not be affected, as well as already scheduled Pods.
This storage class was removed from those servers in T430654: Re-use the ml-cache* machines as dse-k8s-worker nodes supporting the topolvm-balanced storageclass, dse-k8s-eqiad: Reassign dse-k8s-worker1001-4 to the right storageclass and it seems to be applied or broke only after reboot on October 23rd. That explains why it started failing on October 24th.
seems to be a misconfiguration, mentioned servers has device-class capacity, and balanced is available only in dse-k8s-wdqs1004.eqiad.wmnet,dse-k8s-wdqs-test1001.eqiad.wmnet,dse-k8s-worker[1042-1044].eqiad.wmnet
The pods are corresponding to
- dse-k8s-worker1001.eqiad.wmnet
- dse-k8s-worker1004.eqiad.wmnet
Thu, Oct 29
Wed, Oct 28
Fixed permissions on analytics-test-hadoop,
sudo -u hdfs kerberos-run-command hdfs hdfs dfs -chmod 755 /wmf/cache
I tried patching create_easy_dag and it breaks serialisation..
- Review airflow-wikidata: upgrade to Airflow 3.3
- Note what DAGs are enabled: wdqs_external_queries_rollup_daily, table_maintenance_iceberg_monthly
Tue, Oct 27
I found next lines that are suspicious: