In T416170 we concluded to change the Wikibase GraphQL searchItems backend to use MariaDB instead of the current CirrusSearch-based approach. Doing so will allow us to implement additional features like range queries and sorting more easily. The general plan is to store statement data in secondary database tables, similar to what we already do for labels, descriptions and aliases of statements and properties.
Design the schema
See T434870: Create secondary statement store tables.
Populating the tables
We need to create scripts to populate the tables, and then we need to run them on wikidata.org.
Create the scripts
These should probably work similar to the ones for the secondary terms store, e.g. rebuildItemTerms and have similar CLI options:
- from-id/to-id to iterate over
- batch-size
- sleep time
- (?) data/value type(s) to determine what kinds of statements to store
Populate tables on wikidata.org
To fill the new tables with data of all existing items, we need to run the maintenance scripts created in the previous step for wikidata.org. This will take quite some time, depending on the amount of data involved, i.e. whether we want to store all statements, or only a subset. For comparison, initially populating the secondary terms tables took several months.
Updating logic
Once the tables are populated, every relevant item edit should result in an update to the secondary statements store. The exact logic is TBD, but we can get inspiration from how terms are stored in deferred updates.
Migrate GraphQL searchItems to query MariaDB instead
At this point we can create a new search engine implementation that queries MariaDB instead of CirrusSearch. We will probably keep both implementations initially with the new implementation behind a config flag, so that we can control the point at which we switch it over and roll it back if something goes wrong.