Wikidata Analytics Request

This task was generated using the Wikidata Analytics request form. Please use the task template linked on our project page to create issues for the team. Thank you!

Purpose

Please provide as much context as possible as well as what the produced insights or services will be used for.

In T369117: [Analytics] [Request] Generate data processing pipeline for overall Wikidata Co-Editors we're correcting and migrating the monthly Wikidata coeditor metrics so that the process is running on the Hive and Airflow rather than Graphite (at end of life). T369117 will get a new monthly process up and running, and this task will handle the work to populate the created tables with historical data.

Specific Results

Please detail the specific results that the task should deliver.

Historical data for Wikidata coeditors being added into the following two tables:

• wmde.wd_coeditors_by_project_monthly
• wmde.wd_coeditors_distinct_monthly

Desired Outputs

Please list the desired outputs of this task.

• Investigation of the feasible point from which data can be generated
  • Data goes back a very long time, but there will be variations the further back we go, so don't generate data for more than a few years
• Creating a process that can be used to add historical data into the given tables
  • Jupyter based process with data being written to a table in my repo that's then migrated to the production tables and deleted
• Running the process to populate the two tables

Deadline

Please make the time sensitivity of this task clear with a date that it should be completed by. If there is no specific date, then the task will be triaged based on its priority.

04.06.2025

---

Information below this point is filled out by the task assignee.

Assignee Planning

Sub Tasks

A full breakdown of the steps to complete this task.

• Derive base check metric to compare results against
  • Check metric: The numbers generated from the monthly process should be slightly lower than those generated here as some edits will be recorded into the next snapshot
• Write Python/query process to generate old data
• Check numbers from running a historical generation on a current snapshot against the numbers derived from the DAG jobs
  • Will be higher because of edits that are included in next snapshots
  • Will also include edits of deleted pages now
• Generate old data from January 2023 on
• Annotate job query markdown file with a note that the historical data is using a certain snapshot

Estimation

Estimate: 1-2 days
Actual: 1 day

Data

The tables that will be referenced in this task.

• wmf.mediawiki_history

Notes

Things that came up during the completion of this task, questions to be answered and follow up tasks.

• Note