Page MenuHomePhabricator

[2 weeks] Investigate ElasticSearch and alternatives for structured data queries
Closed, ResolvedPublic

Description

As a developer working on Wikibase features, I would like Wikibase to provide a query backend (database) that is suitable for simple structured data queries on statements, e.g. find items with occupation = scientist and field of work chemistry or physics ordered by date of birth. An example of a feature that would benefit from such a database is outlined in T390542 (faceted search).

The existing database technologies we use don't quite fit the bill:

  • the way wikibase stores statements in its primary sql tables is not queryable
  • ElasticSearch + CirrusSearch almost works with the haswbstatement keyword, but there are many limitations e.g. support for logical operators, not supporting comparison operators other than = and the fact that ElasticSearch is made for text search, not structured data
  • WDQS currently doesn't seem performant enough and overly complicated for queries that don't require a graph database

We want to answer the following questions:

  • Can we work around the current limitations of CirrusSearch / ElasticSearch to build a feature like T390542?
  • Are there ways to achieve this using the existing data stores except the Query Service, like MariaDB or ElasticSearch?
  • What alternatives are there e.g. document database like MongoDB?

Event Timeline

Jakob_WMDE renamed this task from Investigate ElasticSearch and alternatives for structured data queries to [2 weeks] Investigate ElasticSearch and alternatives for structured data queries.May 6 2026, 12:28 PM
Jakob_WMDE updated the task description. (Show Details)
Jakob_WMDE moved this task from Polished to Ready for planning on the Wikibase Reuse Team board.

In terms of the feature list, another thing came up that didn't directly belong under https://phabricator.wikimedia.org/T390542 but might be worth checking here which is - querying by entity value but no specific statement property https://phabricator.wikimedia.org/T427888

Change #1301375 had a related patch set uploaded (by Jakob; author: Jakob):

[mediawiki/extensions/Wikibase@master] [DNM] Create tables to store string and entity id statements

https://gerrit.wikimedia.org/r/1301375

Change #1301376 had a related patch set uploaded (by Jakob; author: Jakob):

[mediawiki/extensions/Wikibase@master] [DNM] Store item statements in secondary storage

https://gerrit.wikimedia.org/r/1301376

Change #1301377 had a related patch set uploaded (by Jakob; author: Jakob):

[mediawiki/extensions/Wikibase@master] [DNM] Implement FacetedItemSearchEngine backed by secondary store

https://gerrit.wikimedia.org/r/1301377

Change #1302897 had a related patch set uploaded (by Jakob; author: Jakob):

[mediawiki/extensions/Wikibase@master] [DNM] Store quantity values in secondary storage

https://gerrit.wikimedia.org/r/1302897

Change #1303363 had a related patch set uploaded (by Jakob; author: Jakob):

[mediawiki/extensions/Wikibase@master] [DNM] Enable quantity queries incl ranges

https://gerrit.wikimedia.org/r/1303363

Change #1304062 had a related patch set uploaded (by Jakob; author: Jakob):

[mediawiki/extensions/Wikibase@master] [DNM] Enable sorting by statement value

https://gerrit.wikimedia.org/r/1304062

Change #1305605 had a related patch set uploaded (by Jakob; author: Jakob):

[mediawiki/extensions/Wikibase@master] [DNM] Optimize searchItems SQL query

https://gerrit.wikimedia.org/r/1305605

TL;DR We have aligned on MariaDB as the preferred backend, the main arguments being simplicity, expertise within the team, straightforward infrastructure requirements and lower implementation effort compared to other options.

Detailed investigation results are in this document: https://docs.google.com/document/d/1oDvGnpINmG6rCj9DrCycaq1os-93A-Xen0_Mqy2Fj_A/edit

TL;DR We have aligned on MariaDB as the preferred backend, the main arguments being simplicity, expertise within the team, straightforward infrastructure requirements and lower implementation effort compared to other options.

Detailed investigation results are in this document: https://docs.google.com/document/d/1oDvGnpINmG6rCj9DrCycaq1os-93A-Xen0_Mqy2Fj_A/edit

@Jakob_WMDE / @Ifrahkhanyaree_WMDE / @Silvan_WMDE This document is currently restricted. You are encourage to publish it to a publicly-available page, so (1) people can know why we do not select a NoSQL solution and (2) future developers can know its disadvantage when encoutering a similar situation.

TL;DR We have aligned on MariaDB as the preferred backend, the main arguments being simplicity, expertise within the team, straightforward infrastructure requirements and lower implementation effort compared to other options.

Detailed investigation results are in this document: https://docs.google.com/document/d/1oDvGnpINmG6rCj9DrCycaq1os-93A-Xen0_Mqy2Fj_A/edit

@Jakob_WMDE / @Ifrahkhanyaree_WMDE / @Silvan_WMDE This document is currently restricted. You are encourage to publish it to a publicly-available page, so (1) people can know why we do not select a NoSQL solution and (2) future developers can know its disadvantage when encoutering a similar situation.

Fair point. I created a ticket to document this more publicly in more detail: T435355: Document db technology choice in an ADR.