Page MenuHomePhabricator
Create Task
Maniphest T433138

data-reload: DAG is failing the integrity check
Closed, ResolvedPublic
Actions

Description

The data-reload DAG hasn't gotten past the integrity check since it was launched:
https://airflow-wikidata.wikimedia.org/dags/wdqs_wikidata_data_reload/grid

It looks like some issue with a filename in the PVC:

[2026-07-13, 14:02:00 UTC] {pod_manager.py:358} INFO - [base] ERROR:lib.logging:Could not find file in /mnt/dumpsdata/otherdumps/wikibase/wikidatawiki/20260712/.

Related Objects
Search...

Event Timeline

lerickson created this task.Jul 24 2026, 9:11 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 24 2026, 9:11 PM
BTracy-WMF moved this task from Sprint 07 (2026/07/07 to Sprint 08 (2026/08/04) on the Wikidata Platform Team board.Aug 18 2026, 8:26 PM
BTracy-WMF edited projects, added: Wikidata Platform Team (Sprint 08 (2026/08/04)); removed: Wikidata Platform Team (Sprint 07 (2026/07/07).
trueg changed the task status from Open to In Progress.Sep 4 2026, 11:27 AM
trueg claimed this task.
trueg moved this task from Backlog to In Engineering on the Wikidata Platform Team (Sprint 08 (2026/08/04)) board.
trueg added a parent task: T422179: Set up regular bulk data ingestion and indexing pipeline.Sep 4 2026, 11:29 AM
CodeReviewBot added a project: Patch-For-Review.Sep 4 2026, 3:15 PM

trueg opened https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/2614

WDQS: Fixed wikidata dump/export verification

trueg moved this task from In Engineering to In Code Review on the Wikidata Platform Team (Sprint 08 (2026/08/04)) board.Sep 4 2026, 3:18 PM
CodeReviewBot added a comment.Sep 7 2026, 8:43 AM

trueg merged https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/2614

WDQS: Fixed wikidata dump/export verification

Maintenance_bot removed a project: Patch-For-Review.Sep 7 2026, 9:33 AM
trueg moved this task from In Code Review to Needs Sign-off on the Wikidata Platform Team (Sprint 08 (2026/08/04)) board.Sep 7 2026, 10:26 AM
BTracy-WMF closed this task as Resolved.Wed, Sep 9, 8:46 PM
trueg added a comment.Thu, Sep 10, 12:47 PM

The verification step had multiple issues which now have been resolved:

  • Before the DAG would re-use previously used dump directories by using the end of the schedule cycle as dir name. This resulted in the dump script writing different versions of dump files into the same dir. The verification script correctly flagged this as an issue. Now the DAG uses the start time of the DAG for dump dir naming.
  • With the mentioned change in dump dir, the verification script's hard-coded dump dir detection pattern did not work anymore. It is fixed now, and we plan to make it configurable in the future.
  • The verification script contained a small bug which made it search for dump files in the wrong dir. That is fixed now.
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits