Maniphest T429248

data-reload: munge_dumps task reads TTL data from S3 and writes to HDFS
Closed, ResolvedPublic2 Estimated Story Points
Actions

Assigned To

Authored By

	gmodena
	Jun 15 2026, 7:46 PM

Project Tags

Referenced Files

None

Subscribers

Description

We need to implement a munge_dumps Airflow that invokes

org.wikidata.query.rdf.spark.transform.structureddata.dumps.WikibaseRDFDumpConverter

parametrized to read TTLs from S3 and write to HDFS in the wikidata namespace.

References

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Resolved		gmodena	T431253 [DE2.4.2] WDQS v2 Scaling
		Open		gmodena	T440523 [DE2.6.3] Achieve Launch Readiness for GA
		Stalled		gmodena	T422179 Set up regular bulk data ingestion and indexing pipeline
		Resolved		gmodena	T428237 data-reload: spark processing jobs need to be orchestrated in the wikidata dag.
		Resolved		gmodena	T429248 data-reload: munge_dumps task reads TTL data from S3 and writes to HDFS

Event Timeline

gmodena created this task.Jun 15 2026, 7:46 PM

gmodena mentioned this in T428237: data-reload: spark processing jobs need to be orchestrated in the wikidata dag..

gmodena updated the task description. (Show Details)

gmodena claimed this task.Jun 17 2026, 11:23 AM

gmodena triaged this task as Medium priority.

gmodena edited projects, added: Wikidata Platform Team (Sprint 06 (2026/06/02)); removed: Wikidata Platform Team.

gmodena set the point value for this task to 2.

gmodena moved this task from Backlog to In Engineering on the Wikidata Platform Team (Sprint 06 (2026/06/02)) board.Jun 17 2026, 11:26 AM

gmodena opened https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/2319

Draft: wikidata: orchestrate spark jobs for data preparation and transfer.

gmodena moved this task from In Engineering to In Code Review on the Wikidata Platform Team (Sprint 06 (2026/06/02)) board.Jun 23 2026, 12:35 PM

BTracy-WMF moved this task from Sprint 06 (2026/06/02) to Sprint 07 (2026/07/07 on the Wikidata Platform Team board.Jul 7 2026, 3:16 PM

BTracy-WMF edited projects, added: Wikidata Platform Team (Sprint 07 (2026/07/07); removed: Wikidata Platform Team (Sprint 06 (2026/06/02)).

BTracy-WMF moved this task from Backlog to In Code Review on the Wikidata Platform Team (Sprint 07 (2026/07/07) board.

trueg moved this task from In Code Review to Needs Sign-off on the Wikidata Platform Team (Sprint 07 (2026/07/07) board.Jul 29 2026, 3:27 PM

BTracy-WMF closed this task as Resolved.Aug 3 2026, 8:23 PM