Page MenuHomePhabricator

[Bulk import] Investigate implementation options
Closed, ResolvedPublic

Description

Current Situation:
The initial technical research (T378452) provided an overview of existing bulk import tools and potential strategies for improvements. The next step of the technical research will focus on the REST API.

Goal:

  • Understand the feasibility of implementing a bulk import solution as part of the REST API (endpoint or CLI tool).
  • Gather enough information to inform the design of an MVP and identify open questions.

Event Timeline

adee_wmde changed the task status from Open to In Progress.Feb 13 2025, 1:00 PM
adee_wmde moved this task from Inbox to Sprint-∞ on the Wikibase Suite Team board.
adee_wmde moved this task from Sprint Backlog to Doing on the Wikibase Suite Team (Sprint-∞) board.

Many power users are familiar with OpenRefine. The requirement would be to include the OR reconciliation service bundled into WBS. I'm still not familiar of the practicalities of how to set up it for wikibase edits and for multimedia edits, as I pretend.

T378452 didn't seem to flag up https://gitlab.the-qa-company.com/FrozenMink/batchingestionextension which appears to be the best community API for bulk imports currently (this just came up in the telegram chats).
As has been discovered and forgotten, when going the web API route, one of the biggest improvements is simply being able to POST more than 1 entity at a time to the API.

@Olea indeed, but it was generally decided a few years back that unless you are desperate, direct SQL writing wasn't ideal (it misses a bunch of stuff that you have to run after).

Looking again at the page for RaiseWikibase, it says "Small pages are uploaded at rates of 250-350 wikitexts per second and 220-280 items per second."
https://gitlab.the-qa-company.com/FrozenMink/batchingestionextension looks like it says it can achieve 35-40 for simple entities
With some prototype bulk import web api back a few years ago I managed to get around 150-250 EPM at least, and I imagine that could go faster these days given inner developments of Wikibase (But I do wonder what that numbetr would be!)

Leif_WMDE renamed this task from Bulk import: investigate implementation options to [Bulk import] Investigate implementation options.Jul 22 2025, 8:48 AM

As part of exploring solutions, I'm prototyping a maintenance script which would avoid the overhead of interacting with the REST API. So far I’ve been testing bulk item creation directly with entity store, and looking into ways to batch or parallelize the process to see if that yields better performance.

The starting point for this ticket was using the REST API to get a sense of developer ergonomics. I wrote a simple client that reads JSON files and creates items in a local Wikibase instance via the API. That highlighted the issue of overhead, previously discussed in the first phase of technical research (T378452). To avoid that, I wrote a maintenance script that creates new items directly through entity store. This approach skips some unnecessary checks while preserving data consistency and correctly triggering secondary operations (search index, Recent Changes). The script runs inside the container, so only system administrators with shell access can run it (not to be confused with the "sysops" user group). This fits the assumed main use case for bulk data import, namely initial data import into a fresh instance, or periodic, admin-run updates.
To run the maintenance script, I wrote an item generator that outputs JSON files containing items with a simplified data model. Full benchmarking was out of scope, but I did explore different ways to speed up item creation.

  • ID generation is a major part of new item creation. I added two settings that are meant to keep ID generation from blocking the rest of the saving process:
$wgWBRepoSettings['idGenerator'] = 'mysql-upsert';
$wgWBRepoSettings['idGeneratorSeparateDbConnection'] = true;

I didn't notice an improvement on my machine, but they should help more at scale.
I also looked into the batching ingestion extension, which appears to do bulk ID assignment separately from saving items and skip some of the checks WikiPageEntityStore performs. That could be interesting for future performance comparisons, though I preferred not to reimplement ID generation as part of this prototype for safety and correctness.

  • Most of the performance improvement beyond not having the HTTP overhead comes from running multiple batches in parallel. The maintenance script itself simply reads a JSON file and creates items in a loop, but it's possible to split the input data into several files and invoke the script multiple times. I tried processing partitioned data with & and using parallel. The latter was slightly slower, likely due to output buffering. This should be validated on larger datasets.

Splitting the input into a small number of batches already delivers a significant improvement, but we need to measure multiple runs with different parameters to determine, for example, a balance between the number of jobs and batch size.
Compared to a new REST API endpoint, the maintenance script is easier to develop and deploy, is inherently limited to system admins, avoids HTTP overhead, and has no request size limit or rate limit. On the other hand, we would probably need to implement input deserialization and validation or try to modularize and reuse that part of the REST API. There is also no easy integration with external tools (e.g. Quickstatements) in this case.
For an MVP, I recommend

  • Tuning parameters based on more robust benchmarking.
  • Logging failures per item with meaningful error messages so users can re-run only on those items/batches.
  • Documenting the workflow, which might be initially complex for Wikibase admins (mounting data and running inside the container). Potentially attaching a simple bash script.
  • Implementing data pre-processing to create batch files, adding CSV support.

@adee_wmde: About the ID generation, could be possible to keep ID's exported from a legacy WB instance when importing into a new one? I feel this can be helpful, at least with properties.