Page MenuHomePhabricator

Provide a "wiki farm" abstraction in MediaWiki core
Open, HighPublic

Description

A wiki farm is a group of wikis that work together in some meaningful way (share configuration or user identities, can access each others' database or files etc). It's a vague term as there are many different ways of running wikifarms, with different levels of integration. (It is sometimes also called a wiki cluster or a wiki family, although the latter is also used in a narrower sense, esp. by Pywikibot.) The working definition for this task is that a wikifarm is a group of wikis which can access each others' configuration to some extent, and their infrastructure is set up in such a way that acessing the configuration of another wiki also allows some level of direct data access (they are on the same internal network, etc).

Most large-scale MediaWiki users run wiki farms, because MediaWiki has poor support for some aspects of segmentation of a single wiki (e.g. with respect to language, access control, governance structures, browsing and searching...). But with no "official" support for wiki farms, a wide array of incompatible practices have developed around farms. (See the wiki family manual page for an overview.) Often these are complex and hard to use (including the setup used by Wikimedia); at the same time various wiki farm related concepts have leaked into core, even though they are often conceptually unclear and only work with some specific wikifarm setup (SiteConfiguration is a prime example).

Providing a clean "wiki farm" abstraction in MediaWiki core and reworking the various cross-wiki features (e.g. configuration, interwikis, file repos, "shadow namespace" style features, central user IDs, change tracking) would make MediaWiki less confusing and easier to use and develop, and would enable large MediaWiki users to share work more effectively and cooperate on improving wiki farm management tooling.

Event Timeline

(moved from T224020):
How would this relate to the existing wikifarm related concepts that are already in core? It seems there is already:

  • SiteConfiguration / wgConf
  • WikiMap that builds on top of SiteConfiguration and combines (duplicates?) it with $wgLocalDatabases
  • SiteStore / SiteLookup, i.e. the sites table.

Personally I'm a bit worried that trying to create a new unified replacement would merely extend this list with a fourth option. It already doesn't seem clear (for me at least) how the existing implementations are distributed between use cases—for instance, the sites table only seems to be used by InterwikiLookupAdapter (that is not enabled by default,) WikiMap is extensively used to allow a wiki to push jobs into a job queue of a different wiki (which is a feature that only Wikibase seems to be making use of) etc.

Change 746014 had a related patch set uploaded (by Daniel Kinzler; author: Daniel Kinzler):

[mediawiki/core@master] EXPERIMENT: Add wiki farm support

https://gerrit.wikimedia.org/r/746014

(moved from T224020):
How would this relate to the existing wikifarm related concepts that are already in core? It seems there is already:

  • SiteConfiguration / wgConf
  • WikiMap that builds on top of SiteConfiguration and combines (duplicates?) it with $wgLocalDatabases
  • SiteStore / SiteLookup, i.e. the sites table.

Personally I'm a bit worried that trying to create a new unified replacement would merely extend this list with a fourth option.

The three things you mention all provide information about other wikis in some way. The new mechanism doesn't do that, it merely picks a config file based on some aspect of the request. It's not about connecting the wikis or interactions between them, it's just a convenient mechanism to implement multi-tenancy in MediaWiki.

It would be nice to clean up SiteConfiguration/WikiMap/SiteStore, see T113034. But that is not what this is about.

daniel triaged this task as High priority.Mar 23 2022, 10:24 AM

@daniel Is this meant to block the 1.38 release? It's not clear to me where the urgency for this is coming from.

It mighta also make sense to incubutate this in an extension first, for example, as there are quite a lot of different ways to go about it, and also plenty of documented alternatives on mediawiki.org. I'm also curious to what extent this abstraction layer would be adopted in practice as it seems insufficient for large wiki farms, and small wikis are either not farms or might prefer something simple within LocalSettings.php. We also have $wgConf which provides this functionality already without the need for a separate directory and files etc.

@daniel Is this meant to block the 1.38 release? It's not clear to me where the urgency for this is coming from.

It is not urgent in the sense that anything is blocked on it. The intent is however to make available a number of experimental features related to configuration in 1.39, so people can experiment with it and provide feedback. See https://www.mediawiki.org/wiki/User:DKinzler_(WMF)/Config_in_1.38.

It mighta also make sense to incubutate this in an extension first, for example, as there are quite a lot of different ways to go about it, and also plenty of documented alternatives on mediawiki.org.

That would be nice, but I don't see a way to do this in an extension. Extensions are loaded only after all config has been processed. So they cannot influence the processing of configuration.

I'm also curious to what extent this abstraction layer would be adopted in practice as it seems insufficient for large wiki farms, and small wikis are either not farms or might prefer something simple within LocalSettings.php. We also have $wgConf which provides this functionality already without the need for a separate directory and files etc.

This is driven by the idea that in the small wikis should be able to try out loading settings not from PHP, but from a yaml file. Since you can't do conditionals in a YAML file, a different mechanism for creating a farm is needed.

B that, the intended user base for this feature are docker based development environments and CI suites.

Change 746014 merged by jenkins-bot:

[mediawiki/core@master] Add wiki-farm support

https://gerrit.wikimedia.org/r/746014

Change 776195 had a related patch set uploaded (by Daniel Kinzler; author: Daniel Kinzler):

[mediawiki/core@REL1_38] Add wiki-farm support

https://gerrit.wikimedia.org/r/776195

Change 776196 had a related patch set uploaded (by Daniel Kinzler; author: Daniel Kinzler):

[mediawiki/core@master] WikiFarm: Add missing @since and @unstable tags

https://gerrit.wikimedia.org/r/776196

Change 776196 merged by jenkins-bot:

[mediawiki/core@master] WikiFarm: cleanup comments and dead code.

https://gerrit.wikimedia.org/r/776196

Change 776195 merged by jenkins-bot:

[mediawiki/core@REL1_38] Add wiki-farm support

https://gerrit.wikimedia.org/r/776195

Change 787788 had a related patch set uploaded (by Daniel Kinzler; author: Daniel Kinzler):

[mediawiki/core@master] WikiFarm: remove site detection based on domain name.

https://gerrit.wikimedia.org/r/787788

It is not urgent in the sense that anything is blocked on it. The intent is however to make available a number of experimental features related to configuration in 1.39, so people can experiment with it and provide feedback. See https://www.mediawiki.org/wiki/User:DKinzler_(WMF)/Config_in_1.38.

I appreciate making the functionality available this way before the LTS. I think we need more documentation/publicity about the purpose here -- "Please try this out and give us feedback so we can make sure the LTS is really good".

It is not urgent in the sense that anything is blocked on it. The intent is however to make available a number of experimental features related to configuration in 1.39, so people can experiment with it and provide feedback. See https://www.mediawiki.org/wiki/User:DKinzler_(WMF)/Config_in_1.38.

I appreciate making the functionality available this way before the LTS. I think we need more documentation/publicity about the purpose here -- "Please try this out and give us feedback so we can make sure the LTS is really good".

It's a wiki, feel free to edit :)

Change 787788 merged by jenkins-bot:

[mediawiki/core@master] WikiFarm: remove site detection based on domain name.

https://gerrit.wikimedia.org/r/787788

Also related (about the potentially unwanted coupling between DB name and user-visible wiki ID): T221550: Not possible to set API-reported Wiki ID to anything other than database name