Page MenuHomePhabricator
Create Task
Maniphest T425137

data-reload: set up RDF tables in Ceph, refactor DDL
Closed, DeclinedPublic2 Estimated Story Points
Actions

Description

The import_wikidata_ttl DAG and its associated spark jobs populate some discovery RDF tables that are useful for analytics:

  • discovery.wikibase_rdf_subgraphs
  • discovery.wikibase_rdf

These tables are created via some DDL that is currently inlined into the DAGs.

We should create similar tables in Ceph so that similar data can be produced by the new airflow DAG, and refactor the DDL into the analytics repo.

AC:

  • tables equivalent to the above tables exist in Ceph and are writable by the new wikidata data-reload airflow DAG.
  • DDL has been moved to the analytics repo

Related Objects
Search...

Event Timeline

lerickson created this task.May 1 2026, 6:31 PM
Herald added a subscriber: Aklapper. · View Herald TranscriptMay 1 2026, 6:31 PM
lerickson claimed this task.May 5 2026, 2:38 PM
lerickson edited projects, added: Wikidata Platform Team (Sprint 05 (2026/05/05)); removed: Wikidata Platform Team.May 5 2026, 3:46 PM
DSantamaria added a parent task: T422522: WE2.5.8 WDQS v2 technical build.May 5 2026, 3:58 PM
lerickson added a parent task: T422179: Set up regular bulk data ingestion and indexing pipeline.May 5 2026, 5:39 PM
lerickson removed lerickson as the assignee of this task.May 6 2026, 2:07 PM
lerickson unsubscribed.
lerickson subscribed.
lerickson renamed this task from data-reload: set up RDF tables in CephFS to data-reload: set up RDF tables in CephFS, refactor DDL.May 6 2026, 2:33 PM
lerickson updated the task description. (Show Details)
lerickson set the point value for this task to 2.
lerickson renamed this task from data-reload: set up RDF tables in CephFS, refactor DDL to data-reload: set up RDF tables in Ceph, refactor DDL.May 12 2026, 12:52 PM
lerickson updated the task description. (Show Details)
DSantamaria moved this task from Sprint 05 (2026/05/05) to Backlog on the Wikidata Platform Team board.Jun 2 2026, 3:18 PM
DSantamaria edited projects, added: Wikidata Platform Team; removed: Wikidata Platform Team (Sprint 05 (2026/05/05)).
gmodena closed this task as Declined.Jul 6 2026, 9:28 AM
gmodena subscribed.

Decline since won't be using ceph/s3 backed tables.

DDL work is already covered in T429254: data-reload: keep data pipeline tables DDL under version control.

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits