Page MenuHomePhabricator
Create Task
Maniphest T402529

Airflow instance 'airflow-test-k8s' looses all XML dumps DAGs intermittently
Closed, ResolvedPublic
Actions

Description

While checking progress on the XML enwiki dump run, I got a DAG “mediawiki_dumps_sql_xml_large_a_to_z_full” seems to be missing from DagBag. error from Airflow:

Screenshot 2025-08-12 at 8.50.28 AM.png (728×1 px, 143 KB)

@BTullis mentions that:

We have seen it before and I think that it is related to DAG parsing timeouts. We have even seen it on airflow-main. Should be largely harmless, as tasks a re-adopted, but it would be better if we could stop it from happening.

I was just able to reproduce this as of the creation of this ticket. All XML dump related DAGs were lost for about a minute. All other DAGs seems to be fine.

Slack thread.

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
airflow-test-k8s: increase DAG file parsing intervaloperations/deployment-chartsmaster+4 -5
mediawiki-dumps-legacy: adapt RBAC to a recent apache-airflow-providers-cncf-kubernetes upgradeoperations/deployment-chartsmaster+4 -2
Customize query in gerrit
Related Changes in GitLab:
TitleReferenceAuthorSource BranchDest Branch
Draft: Some simple performance improvements for mediawiki_sql_xml_dumps.pyrepos/data-engineering/airflow-dags!1634amastilovicT402529_mediawiki_sql_dumps_perf_improvmain
Draft: test_k8s/dumps/sql-xml: optimize the DAG file execution timerepos/data-engineering/airflow-dags!1632brouberolT402529-single-filemain
test_k8s/dumps/sql_xml: code generate each DAG to free scheduler resourcesrepos/data-engineering/airflow-dags!1625brouberolT402529main
Customize query in GitLab

Event Timeline

xcollazo created this task.Aug 21 2025, 2:00 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 21 2025, 2:00 PM
xcollazo updated the task description. (Show Details)Aug 21 2025, 2:01 PM
xcollazo added a comment.Aug 21 2025, 2:28 PM

Another repro just now.

xcollazo triaged this task as Medium priority.Aug 21 2025, 2:33 PM

And again. :(

Bumping priority.

CodeReviewBot added a project: Patch-For-Review.Aug 22 2025, 9:43 AM

brouberol opened https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/1625

test_k8s/dumps/sql_xml: code generate each DAG to free scheduler resources

brouberol changed the task status from Open to In Progress.Aug 22 2025, 3:15 PM
brouberol claimed this task.
brouberol moved this task from Backlog - project to Needs Review on the Data-Platform-SRE (2025.08.16 - 2025.09.05) board.
brouberol edited projects, added Data-Platform-SRE (2025.08.16 - 2025.09.05); removed Data-Platform-SRE.
brouberol added a comment.Aug 22 2025, 3:17 PM

I've submitted an MR with a large rewrite of the DAG file. Instead of having a single DAG file iterating over partial/full, large/regular, all wiki buckets and each wiki in each bucket. we codegen one dag file for each of the 20 SQL/XML dump DAGs we run.

IMHO this causes an increase of the maintenance burden as the complexity of the overall setup increases, but it dramatically improves how Airflow behaves, alongside several code improvements I found through profiling the generated dag files.

CodeReviewBot added a comment.Aug 25 2025, 6:47 AM

brouberol opened https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/1632

Draft: test_k8s/dumps/sql-xml: optimize the DAG file execution time

gerritbot added a comment.Aug 25 2025, 7:36 AM

Change #1181519 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/deployment-charts@master] mediawiki-dumps-legacy: adapt RBAC to a recent apache-airflow-providers-cncf-kubernetes upgrade

https://gerrit.wikimedia.org/r/1181519

gerritbot added a comment.Aug 25 2025, 1:59 PM

Change #1181519 merged by Brouberol:

[operations/deployment-charts@master] mediawiki-dumps-legacy: adapt RBAC to a recent apache-airflow-providers-cncf-kubernetes upgrade

https://gerrit.wikimedia.org/r/1181519

CodeReviewBot added a comment.Aug 25 2025, 4:38 PM

amastilovic opened https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/1634

Some simple performance improvements for mediawiki_sql_xml_dumps.py

gerritbot added a comment.Sep 2 2025, 12:38 PM

Change #1184064 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/deployment-charts@master] airflow-test-k8s: increase DAG file parsing interval

https://gerrit.wikimedia.org/r/1184064

gerritbot added a comment.Sep 2 2025, 12:42 PM

Change #1184064 merged by Brouberol:

[operations/deployment-charts@master] airflow-test-k8s: increase DAG file parsing interval

https://gerrit.wikimedia.org/r/1184064

CodeReviewBot added a comment.Sep 2 2025, 12:45 PM

brouberol merged https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/1625

test_k8s/dumps/sql_xml: code generate each DAG to free scheduler resources

brouberol added a comment.Sep 2 2025, 1:04 PM

The change had a positive impact on both scheduler CPU usage as well as DAG parsing time. The DAGs are no longer disappearing from the UI.

image.png (610×2 px, 293 KB)
image.png (546×2 px, 158 KB)

brouberol closed this task as Resolved.Sep 2 2025, 1:04 PM
brouberol moved this task from Needs Review to Done on the Data-Platform-SRE (2025.08.16 - 2025.09.05) board.
CodeReviewBot added a comment.Sep 2 2025, 3:50 PM

brouberol closed https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/1632

Draft: test_k8s/dumps/sql-xml: optimize the DAG file execution time

CodeReviewBot added a comment.Sep 2 2025, 8:17 PM

amastilovic closed https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/1634

Draft: Some simple performance improvements for mediawiki_sql_xml_dumps.py

Maintenance_bot removed a project: Patch-For-Review.Sep 2 2025, 8:31 PM
Gehel moved this task from Done to Reported on the Data-Platform-SRE (2025.08.16 - 2025.09.05) board.Sep 5 2025, 12:46 PM
Gehel added a project: Essential-Work.Sep 5 2025, 3:02 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits