Page MenuHomePhabricator

[X1][Wikibase] Set up virtual domains for entity usage db connection. (FM) (๐Ÿ†|๐ŸฆŠ)
Open, Needs TriagePublic

Description

We decided to use virtual databases.

Open questions:

  • Is sub-domains included?

In order to enable to optimize the database infrastructure usage by moving wbc_entity_usage tables of all wikis to the extension database cluster (x1), Wikibase software should be able to work with multiple database configurations ("get data from multiple database servers")

Notes:

  • Mediawiki's virtual database domains is a functionality that provides a helpful way of abstracting database configuration - let's use this. Similar to the example below
  • Relatively similar work has been done by WMDE a few years ago when term-related database tables of wikidata have been moved to another database cluster. See work in T351802: Wikibase: Introduce separate database configuration for term store / I328c1149466761 for an idea.
  • Before the actual move of wbc_entity_usage happens, the configuration should point to the current "main" database configuration, to keep the software work as before.

OOS:

  • Adjusting DB queries. See ticket below.

Note: parallel to this ticket, database queries that read data from wbc_entity_usage table and some other Mediawiki table (join) need to be adjusted as those queries would no longer work - this is extracted as a subticket

Event Timeline

Very likely the existing database queries that are touching tables that would be now in different cluster - which I understand wouldn't work anymore
Those queries should probably be changed to get the same data in a way that work in the new setting

Example queries

  1. Getting pages (with usage details or without) that make use of a particular Wikidata items, e.g.

https://github.com/wikimedia/mediawiki-extensions-Wikibase/blob/5c12f7e7ea876d4577da4a5726de85dda1e169ba/client/includes/Specials/SpecialEntityUsage.php#L138-L154
https://github.com/wikimedia/mediawiki-extensions-Wikibase/blob/5c12f7e7ea876d4577da4a5726de85dda1e169ba/client/includes/Api/ApiListEntityUsage.php#L171-L200

A fairly naive way of rewriting those queries would be to first fetch page IDs (and aspects if relevant) from wbc_entity_usage and then query page title and names from page using a list of IDs (check for sensible size of the list, might need to batch it)

SuzanneWood-WMDE renamed this task from Adjust Wikibase logic to have a separate database configuration for the entity usage storage to [X1] Adjust Wikibase logic to have a separate database configuration for the entity usage storage.May 20 2026, 12:00 PM
SuzanneWood-WMDE renamed this task from [X1] Adjust Wikibase logic to have a separate database configuration for the entity usage storage to [X1] Adjust Wikibase logic to have a separate database configuration for the entity usage storage (parallel).Jun 11 2026, 12:37 PM
SuzanneWood-WMDE updated the task description. (Show Details)
SuzanneWood-WMDE updated the task description. (Show Details)
SuzanneWood-WMDE updated the task description. (Show Details)
Lucyfediachambers renamed this task from [X1] Adjust Wikibase logic to have a separate database configuration for the entity usage storage (parallel) to [X1][Wikibase] Adjust Wikibase logic to have a separate database configuration for the entity usage storage (parallel).Jun 24 2026, 3:12 PM
Lucyfediachambers renamed this task from [X1][Wikibase] Adjust Wikibase logic to have a separate database configuration for the entity usage storage (parallel) to [X1][Wikibase] Set up virtual domains for entity usage db connection..Jun 25 2026, 11:27 AM
Lucyfediachambers renamed this task from [X1][Wikibase] Set up virtual domains for entity usage db connection. to ๐Ÿงน [X1][Wikibase] Set up virtual domains for entity usage db connection..
Lucyfediachambers updated the task description. (Show Details)
Lucyfediachambers renamed this task from ๐Ÿงน [X1][Wikibase] Set up virtual domains for entity usage db connection. to ๐Ÿงน [X1][Wikibase] Set up virtual domains for entity usage db connection. (FM).Jun 25 2026, 11:30 AM
Lucyfediachambers renamed this task from ๐Ÿงน [X1][Wikibase] Set up virtual domains for entity usage db connection. (FM) to ๐Ÿงน [X1][Wikibase] Set up virtual domains for entity usage db connection. (FM) (๐Ÿ†).Fri, Jul 17, 8:14 AM

Proof of concept is done - Next check-in/status update Friday 24.07.

Change #1314026 had a related patch set uploaded (by Seanleong-wmde; author: Seanleong-wmde):

[mediawiki/extensions/Wikibase@master] Add virtual-wikibase-entityusage virtual domain

https://gerrit.wikimedia.org/r/1314026

Lucyfediachambers renamed this task from ๐Ÿงน [X1][Wikibase] Set up virtual domains for entity usage db connection. (FM) (๐Ÿ†) to [X1][Wikibase] Set up virtual domains for entity usage db connection. (FM) (๐Ÿ†).Thu, Jul 23, 12:00 PM
seanleong-WMDE renamed this task from [X1][Wikibase] Set up virtual domains for entity usage db connection. (FM) (๐Ÿ†) to [X1][Wikibase] Set up virtual domains for entity usage db connection. (FM) (๐Ÿ†|๐ŸฆŠ).Mon, Jul 27, 8:13 AM

Hi @Ladsgroup! I have some questions regarding the difference between getting the connection to the terms table and the entity usage table.

  1. What is the reason the connection to the terms table in RepoDomainTermsDb.php uses RepoDomainDb->connections() directly, whereas the connection to the entity usage table in "DirectSqlStore.php" uses clientDomainDb->sessionConsistentConnections()? Specifically, the reason for the difference between connections() and sessionConsistentConnections() in both cases?
  1. If it's needed, perhaps a similar logic to the sessionConsistentConnectionManager.php can be implemented in the LBFactory.php when returning a primary database to the X1 cluster?
  1. As I am also referring to this patch, for example in the VirtualTermsDomainDb.php, does it make sense that instead of passing and accessing an ILBFactory directly, a ClientDomainDbFactory->newVirtualDomainDb( $domain ): VirtualDomainDb is created instead? This way, we can inject the client VirtualDomainDb directly and also have access to sessionConsistentConnectionManager.
  1. I think you have heard about this already, but is there a way to test out the implementation with the virtual cluster rather than creating a table locally and going through the use cases manually?

Thank you!

Update: questions answered from communication between Amir and Jakob, thanks!