Wikidata Analytics Request

This task was generated using the Wikidata Analytics request form. Please use the task template linked on our project page to create tasks for the team. Thank you!

Purpose

Please provide as much context as possible as well as what the produced insights or services will be used for.

Specific Results

Please detail the specific results that the task should deliver.

The current process of exporting data to the published datasets directories has gotten significantly more complex in recent months and has been dramatically slowing down the full delivery of DAGs by WMDE SWE Analytics. The old practice was that all data should be exported as CSVs to these directories for all DAGs. We've made the decision to not do this anymore.

Desired Outputs

Please list the desired outputs of this task.

• Rework hql test script generation process to work with new query sets
• Remove export step from all wmde instance DAGs
• Delete all data from prior exports in the published datasets directories
• Create a task to look into the need for other WMDE data that's in the published datasets directories
  • T393446

Deadline

Please make the time sensitivity of this request clear with a date that it should be completed by. If there is no specific date, then the task will be triaged based on its priority.

30.4.2025

---

Information below this point is filled out by the task assignee.

Assignee Planning

Sub Tasks

A full breakdown of the steps to complete this task.

• Update test generation script to not require an export file
• Remove export steps from all DAGs - data-engineering/airflow-dags#1187
• Rewrite DAG tests to not include the export steps - data-engineering/airflow-dags#1187
• Delete the data on HDFS
• Minor exploration of other WMDE data in the published datasets directories
  • Lots of data still present from prior Analytics team
  • Create a task for deleting data that might need to be
  • T393446

Estimation

Estimate: 1 day
Actual: 1 day over a longer period

Notes

Things that came up during the completion of this task, questions to be answered and follow up tasks.

• Note