Page MenuHomePhabricator

Migrate GraphQL item search to MariaDB
Open, Needs TriagePublic

Description

In T416170 we concluded to change the Wikibase GraphQL searchItems backend to use MariaDB instead of the current CirrusSearch-based approach. Doing so will allow us to implement additional features like range queries and sorting more easily. The general plan is to store statement data in secondary database tables, similar to what we already do for labels, descriptions and aliases of statements and properties.

Design the schema

See T434870: Create secondary statement store tables.

Populating the tables

We need to create scripts to populate the tables, and then we need to run them on wikidata.org.

Create the scripts

These should probably work similar to the ones for the secondary terms store, e.g. rebuildItemTerms and have similar CLI options:

  • from-id/to-id to iterate over
  • batch-size
  • sleep time
  • (?) data/value type(s) to determine what kinds of statements to store

Populate tables on wikidata.org

To fill the new tables with data of all existing items, we need to run the maintenance scripts created in the previous step for wikidata.org. This will take quite some time, depending on the amount of data involved, i.e. whether we want to store all statements, or only a subset. For comparison, initially populating the secondary terms tables took several months.

Updating logic

Once the tables are populated, every relevant item edit should result in an update to the secondary statements store. The exact logic is TBD, but we can get inspiration from how terms are stored in deferred updates.

Migrate GraphQL searchItems to query MariaDB instead

At this point we can create a new search engine implementation that queries MariaDB instead of CirrusSearch. We will probably keep both implementations initially with the new implementation behind a config flag, so that we can control the point at which we switch it over and roll it back if something goes wrong.

Event Timeline

Following up on a chat with @Jakob_WMDE about this task, some thoughts on two of the open points in the description: which statements to store ("all statements, or only a subset"), and the updating logic (currently TBD).

On the subset question, one natural candidate subset is "truthy" statements, i.e. the best-rank statements per property that wdt: exposes in RDF, since many consumers only ever want those. But truthiness is awkward for a table like this, whether it is used to pick the stored subset or filtered at query time, because it is not a property of a single row but of the (subject, property) group: preferred if any, else normal, deprecated never (the StatementList::getBestStatements() semantics). No encoding of rank on a single row can make that a clean WHERE clause. Suggestion: store rank for every statement (the prototype already does, see the schema and the writer) and have the updating logic maintain a denormalized is_best flag, recomputed for the affected (subject, property) group on write. RDF output already ships this dual encoding (wdt: triples plus wikibase:BestRank), and both RDF builders carry a FIXME about each recomputing best-per-property independently (TruthyStatementRdfBuilder, FullStatementRdfBuilder), so precomputing it once at write time has precedent and a known pain point it would relieve.

On how much of the subset question needs answering up front: mostly none of it. The population scripts are re-runnable, so "which statements to store" (truthy only, certain value or data types, items only) can start narrow and be widened later, as long as the schema does not change. The updating logic is the one decision that does not backfill: with delete-and-reinsert (what the prototype currently does: SecondaryStatementStoreWriter.php), anything about when or how a row changed is unrecoverable after the fact. So of the TBDs in the description, is that the one worth settling deliberately in the acceptance criteria?