[Analytics] Baseline metrics for data dumps
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	Ifrahkhanyaree_WMDE
	Jul 17 2025, 8:40 AM

Description

Wikidata Analytics Request

This task was generated using the Wikidata Analytics request form. Please use the task template linked on our project page to create issues for the team. Thank you!

Purpose

We need the following baseline metrics for data dumps per month:

# downloads (full v partial across the different types)
# downloads per unique ip x user agent
# unique user agents
Size of a full data dump in GB

Desired Outputs

Please list the desired outputs of this task.

Baseline metrics for Wikidata data dumps

Deadline

Please make the time sensitivity of this task clear with a date that it should be completed by. If there is no specific date, then the task will be triaged based on its priority.

04.08.2025

Information below this point is filled out by the task assignee.

Assignee Planning

Sub Tasks

A full breakdown of the steps to complete this task.

Explicitly define the metrics needed
Write all create table queries for the DAG jobs
Write all the computation queries for the DAGs
Test query processes in local DB schema
Write DAG(s)
Test DAG(s)
Deploy DAG(s)

Estimation

Estimate: 2-3 days
Actual:

Data

The tables that will be referenced in this task.

link_to_table

Notes

Things that came up during the completion of this task, questions to be answered and follow up tasks.

Note

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Resolved		Ifrahkhanyaree_WMDE	T399808 [Analytics] Baseline metrics for data dumps
		Resolved		Ifrahkhanyaree_WMDE	T399809 [Analytics] Superset dashboard for data dumps

Event Timeline

Ifrahkhanyaree_WMDE created this task.Jul 17 2025, 8:40 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 17 2025, 8:40 AM

Ifrahkhanyaree_WMDE added a subtask: T399809: [Analytics] Superset dashboard for data dumps.Jul 17 2025, 8:42 AM

Ifrahkhanyaree_WMDE added a project: Wikidata.

Ifrahkhanyaree_WMDE renamed this task from Baseline metrics for data dumps to [Analytics] Baseline metrics for data dumps.Jul 17 2025, 8:54 AM

AndrewTavis_WMDE moved this task from Incoming to Backlog on the Wikidata Analytics board.Jul 17 2025, 12:27 PM

AndrewTavis_WMDE claimed this task.Aug 22 2025, 1:00 PM

AndrewTavis_WMDE moved this task from Backlog to Kanban on the Wikidata Analytics board.

AndrewTavis_WMDE edited projects, added Wikidata Analytics (Kanban); removed Wikidata Analytics.

AndrewTavis_WMDE moved this task from Incoming to To-Do on the Wikidata Analytics (Kanban) board.

AndrewTavis_WMDE moved this task from To-Do to In Progress on the Wikidata Analytics (Kanban) board.Aug 22 2025, 1:14 PM

AndrewTavis_WMDE updated the task description. (Show Details)Aug 25 2025, 9:55 AM

AndrewTavis_WMDE mentioned this in T403159: Airflow processes to import dump logs and generate monthly metrics.Aug 28 2025, 10:50 AM

AndrewTavis_WMDE moved this task from In Progress to To-Do on the Wikidata Analytics (Kanban) board.Sep 4 2025, 3:35 PM

Hallo halloo, just checking - would you have an idea of when I could have these numbers by?

Hellooo 👋 We just heard back that T403159 will still be a bit, so this will need to be a recurring task that done manually. I'd been hoping that the work for this could be a DAG. I'll try to get the notebook set up and get the first few months of data by EOW or early next week at the latest as I've been able to make a lot of progress on the ones that jumped ahead of this one :)

That sounds good! If I get some data for the last few months of this year, then I can use that to come up with a KR for next year. After that there's no urgency to this ticket, so no worries

AndrewTavis_WMDE moved this task from To-Do to In Progress on the Wikidata Analytics (Kanban) board.Oct 6 2025, 12:09 PM

AndrewTavis_WMDE moved this task from In Progress to In Review on the Wikidata Analytics (Kanban) board.Oct 8 2025, 11:46 AM

Moving this into In Review :)

As of now we can't do a DAG for this process, so I'll be manually running a process defined in T399808_wd_dump_metrics.ipynb. We need to get the logs from stat1011, and then a process is ran to generate the data for the wmde.wd_dump_request_metadata_monthly and wmde.wd_dump_metrics_monthly tables. This information has since also been displayed on the wmde_wd_dump_metrics_monthly Superset dashboard.

please review and close if all is good :)

Ifrahkhanyaree_WMDE closed this task as Resolved.Fri, Nov 14, 10:47 AM

Ifrahkhanyaree_WMDE closed subtask T399809: [Analytics] Superset dashboard for data dumps as Resolved.Tue, Nov 18, 7:44 AM

[Analytics] Baseline metrics for data dumpsClosed, ResolvedPublicActions