FY2025-26 WE 6.4.1: Move links tables of commons to a dedicated cluster
Open, MediumPublic
Actions

Assigned To

Authored By

	Ladsgroup
	Jul 4 2025, 12:04 PM

Description

This is part of annual plan for FY 25-26. Commons database has grown to an extremely large size (2TB) and we have no choice but to split it. The most cost-benefit efficient way is to move the links tables to a dedicated cluster (x4) which would cut its size to half.

These tables must be moved to x4:

linktarget
externallinks
pagelinks
templatelinks
categorylinks
collation
imagelinks
globalimagelinks
iwlinks
existencelinks

Maybe:

langlinks

This can be done via introducing a new core virtual domain (virtual-links) and then migrating queries. Any query that join with page or other tables must be split to be two queries hitting different connections and if not possible, it should be done in a different way (e.g. T309738: Move MediaWiki QueryPages computation to Hadoop) or gets disabled in commons.

We need to set up back ups and wikireplicas too.

For now, we should assume page and redirect tables will exist in both databases. Depending on how many cases directly join and not easy to split, we could consider having those tables being shared.

Related Objects
Search...

Status	Subtype	Assigned	Task
Open		None	T343131 Commons database is growing way too fast
Open		Ladsgroup	T398709 FY2025-26 WE 6.4.1: Move links tables of commons to a dedicated cluster
Resolved		Ladsgroup	T398829 Migrate queries of externallinks to virtual-links
Open		Ladsgroup	T400824 Prepare globalimagelinks for s4 split
Resolved		Zabe	T400169 Convert GlobalUsage to virtual domains
Resolved		Zabe	T400883 Migrate queries of templatelinks to virtual domain
Resolved		Zabe	T402354 Migrate queries of imagelinks to virtual-links
Resolved		Zabe	T404711 Migrate existencelinks to virtual domain
Stalled		• Marostegui	T404715 Setup fake x4 section
Resolved		• Marostegui	T406550 Productionize db126[0-3]
Resolved		• Marostegui	T406551 Productionize db224[5-8]
Resolved		Zabe	T405809 Migrate iwlinks to virtual domains
Open		Zabe	T405811 Migrate pagelinks to virtual domain
Open		Zabe	T405812 Migrate categorylinks to virtual domain
Open		Ladsgroup	T408137 Build a write duplicator system in mediawiki core
Open		Zabe	T309738 Move MediaWiki QueryPages computation to Hadoop
Resolved	PRODUCTION ERROR	Scott_French	T408530 MediaWiki periodic job update-special-pages-s3 failed
Resolved	PRODUCTION ERROR	Zabe	T408801 Expectation (readQueryRows <= 10000) by MediaWiki\Actions\ActionEntryPoint::execute not met (caused by RestrictionStore)
Open		None	T411577 Make ChangesListQuery compatible with commons split

Event Timeline

Ladsgroup created this task.Jul 4 2025, 12:04 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 4 2025, 12:04 PM

Ladsgroup triaged this task as Medium priority.Jul 4 2025, 12:05 PM

Ladsgroup moved this task from Inbox to Epic - Database on the Data-Persistence board.

Aklapper renamed this task from WE 6.4.1: Move links tables of commons to a dedicated cluster to FY2025-26 WE 6.4.1: Move links tables of commons to a dedicated cluster.Jul 4 2025, 1:35 PM

Ladsgroup updated the task description. (Show Details)Jul 7 2025, 8:23 PM

Ladsgroup updated the task description. (Show Details)Jul 11 2025, 12:04 PM

Zabe subscribed.Jul 11 2025, 10:01 PM

For globalimagelinks we may consider make it functionally separate (using a second virtual domain, do not use core linktarget, etc) given T241053#10001833.
We may consider also moving links table in Wikidata. pagelinks is currently the second largest table after terms being moved to a dedicated cluster.

Bugreporter mentioned this in T383507: Change data type of namespace fields from int to smallint or mediumint.Jul 12 2025, 7:26 PM

Don-vip awarded a token.Jul 27 2025, 7:28 PM

Don-vip subscribed.

Ladsgroup added a project: OKR-Work.Jul 31 2025, 8:41 PM

Ladsgroup updated the task description. (Show Details)Jul 31 2025, 10:19 PM

Maryann-Onyinye subscribed.Aug 3 2025, 1:28 PM

Ladsgroup closed subtask T400883: Migrate queries of templatelinks to virtual domain as Resolved.Aug 13 2025, 8:50 PM

Ladsgroup mentioned this in T399458: Move category changes from RecentChanges/Watchlist to separate Special page.Aug 19 2025, 10:44 AM

Ladsgroup closed subtask T398829: Migrate queries of externallinks to virtual-links as Resolved.Aug 19 2025, 1:04 PM

Ladsgroup updated the task description. (Show Details)Aug 26 2025, 11:30 PM

Ottomata subscribed.Sep 2 2025, 6:13 PM

Zabe updated the task description. (Show Details)Sep 16 2025, 12:30 PM

Krinkle mentioned this in T405005: Implement mechanism to exclude a domain from externallinks database (LinkSearch).Sep 18 2025, 4:24 PM

Zabe closed subtask T404711: Migrate existencelinks to virtual domain as Resolved.Sep 28 2025, 12:14 PM

Zabe closed subtask T405809: Migrate iwlinks to virtual domains as Resolved.Sep 28 2025, 4:12 PM

Novem_Linguae subscribed.Oct 1 2025, 5:58 PM

Msz2001 subscribed.Oct 6 2025, 7:08 AM

Zabe reopened subtask T400883: Migrate queries of templatelinks to virtual domain as Open.Oct 11 2025, 3:04 PM

ReaperDawn subscribed.Oct 20 2025, 7:58 AM

Doesn't this require communication towards the community similar to how the x3 change was announced for the Wikidata terms tables ?

In T398709#11293949, @TheDJ wrote:

Doesn't this require communication towards the community similar to how the x3 change was announced for the Wikidata terms tables ?

Yup but we are at least several months away from the split actually happening. There are a lot of complex issues that need to be solved e.g. how to replicate the page table from s4 to x4 (and the consolidation mechanism) and so on.

Wellverywell subscribed.Oct 21 2025, 6:35 PM

• Marostegui changed the status of subtask T404715: Setup fake x4 section from Open to Stalled.Oct 22 2025, 11:38 AM

• Marostegui mentioned this in T401295: Decide how to use the new clouddb hosts (clouddb102[2-5]).Oct 22 2025, 1:14 PM

Ladsgroup added a subtask: T408137: Build a write duplicator system in mediawiki core.Oct 23 2025, 4:06 PM

I noticed in a few commits that there is not just virtual-links but also a more narrow virtual-categorylinks. Is there more detail somehere on how that is meant to work exactly? The current plan describes a dedicated cluster for the link tables, which is easy to reason about and there's the task listing which tables are available to query/join there (with page+redirect earmarked as special TBD).

A domain like virtual-categorylinks suggests more decoupling, but I couldn't find any discussion or docs on what it is meant to be used for or what tables one we assume to exist there? Some of the queries using this join against linktarget, for example.

In T398709#11308044, @Krinkle wrote:

I noticed in a few commits that there is not just virtual-links but also a more narrow virtual-categorylinks. Is there more detail somehere on how that is meant to work exactly? The current plan describes a dedicated cluster for the link tables, which is easy to reason about and there's the task listing which tables are available to query/join there (with page+redirect earmarked as special TBD).

A domain like virtual-categorylinks suggests more decoupling, but I couldn't find any discussion or docs on what it is meant to be used for or what tables one we assume to exist there? Some of the queries using this join against linktarget, for example.

The idea was to mostly ease the migration of read queries and make sure they are correctly moved (specially for the test setup). Once we are done, all will be consolidated into virtual-links.