Maniphest T423012

Use dumps
Closed, ResolvedPublic13 Estimated Story Points
Actions

Assigned To

Authored By

	Danya
	Apr 11 2026, 7:52 PM

Project Tags

Referenced Files

None

Subscribers

Description

QLever having shown its limits for the sizes of requests we need to perform, we must use Wikidata dumps instead. The most efficient way looks to be the "truthy" N-Triples dumps (.nt files).

Resources:

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Resolved		Danya	T427546 [VERSION] 0.2.0
		Resolved	Feature	Danya	T422342 Retro-active statistics
		Resolved		Danya	T427430 [VERSION] 0.1.0
		Resolved		Danya	T422343 Integrate Denelezh and merge stats tables
		Resolved		Danya	T423012 Use dumps

Event Timeline

Danya created this task.Apr 11 2026, 7:52 PM

Danya set the point value for this task to 5.May 1 2026, 10:45 AM

Wikidata Toolkit may be useful here. This is what Humaniki was using.

Danya claimed this task.May 7 2026, 6:21 PM

Danya triaged this task as High priority.May 13 2026, 7:08 PM

Danya added a parent task: T422343: Integrate Denelezh and merge stats tables.

Danya moved this task from Backlog to [deleted] on the Tool-wiki-gender-stats board.

Danya renamed this task from Use dumps? to Use dumps.May 13 2026, 7:11 PM

Danya updated the task description. (Show Details)

Danya changed the point value for this task from 5 to 6.

Danya added a subtask: T425489: Migrate site to Cloud VPS?.May 13 2026, 7:14 PM

Danya edited subtasks, added: T425991: Investigate best solution between ToolsDB and self-hosted database; removed: T425489: Migrate site to Cloud VPS?.May 13 2026, 7:18 PM

Danya changed the task status from Open to In Progress.May 14 2026, 8:38 AM

Danya moved this task from [deleted] to Doing on the Tool-wiki-gender-stats board.

Danya removed a subtask: T425991: Investigate best solution between ToolsDB and self-hosted database.

Danya changed the point value for this task from 6 to 13.May 14 2026, 8:45 AM

In the end I used the JSON dumps as their structure is stable and easier to parse.

Implemented in T422343 but needs to be tested in prod.

Great news, Wikidata entities dumps can be accessed directly from toolforge at /public/dumps/public/wikidatawiki/entities/, no download needed.

danyya opened https://gitlab.wikimedia.org/toolforge-repos/humaniki/-/merge_requests/61

Draft: Resolve T423012 "Use dumps"

danyya merged https://gitlab.wikimedia.org/toolforge-repos/humaniki/-/merge_requests/61

Resolve T423012 "Use dumps"

Danya closed this task as Resolved.May 28 2026, 3:09 PM

Danya moved this task from Doing to Done on the Tool-wiki-gender-stats board.

Maintenance_bot removed a project: Patch-For-Review.May 28 2026, 3:30 PM