Page MenuHomePhabricator

Add support for multi-DB testing to MediaWiki
Open, Needs TriagePublic

Description

MediaWiki has several features that make use of multiple database connections. This includes features that need to query data across multiple wikis (such as CheckUser-GlobalContributions), that write data to a foreign wiki (such as Special:UserRight's interwiki mode the Stewards use) or that store data at the x1 cluster (Notifications (Echo), GrowthExperiments, StructuredDiscussions, MediaWiki-extensions-ReadingLists, ...). Just like any other features, they benefit from being covered by appropriate tests. Unfortunately, features depending on multiple databases cannot be covered by integration tests at all. This is because CI only runs in a single wiki mode.

In the past, the lack of support for multi-DB testing caused significant outages. For example, Special:UserRights's interwiki mode broke down for ~three consecutive trains, as neither developers nor CI had multi DB setup enabled (and tested for).

This task tracks adding the support for that testing to the CI infrastructure.

Event Timeline

For T342763, @Urbanecm proposed a patch to Quibble to have it spin up a second database ( https://gerrit.wikimedia.org/r/c/integration/quibble/+/949986 ).

If at all possible, I'd like to avoid adding that to Quibble in favor of using MediaWiki system to simulate a secondary database using a prefix or cloning the DB. I apparently gave it a try at https://gerrit.wikimedia.org/r/c/mediawiki/core/+/951950/1/tests/phpunit/includes/specials/SpecialUserRightsTest.php :

$primaryDB = wfGetDB( DB_PRIMARY );
$externalDBname = $primaryDB->getDBname() . '-externaldb';

$dbClone = new CloneDatabase( $primaryDB, [
        // User related tables
        'user', 'user_groups', 'user_properties', 'actor',
    ],
    'externaldb',
    $primaryDB->tablePrefix(), // unittest_
);
$dbClone->useTemporaryTables( true );
$dbClone->cloneTableStructure();

But that leads to a mysterious failure:

Wikimedia\Rdbms\DBUnexpectedError:

Cannot directly change the selected DB domain; any underlying connection handle is owned by a LoadBalancer instance and possibly shared with other callers. LoadBalancer automatically manages DB domain re-selection of unused handles.

For T342763, @Urbanecm proposed a patch to Quibble to have it spin up a second database ( https://gerrit.wikimedia.org/r/c/integration/quibble/+/949986 ).

If at all possible, I'd like to avoid adding that to Quibble in favor of using MediaWiki system to simulate a secondary database using a prefix or cloning the DB.

@aaron, do you think this might be attainable in CI?

For T342763, @Urbanecm proposed a patch to Quibble to have it spin up a second database ( https://gerrit.wikimedia.org/r/c/integration/quibble/+/949986 ).

If at all possible, I'd like to avoid adding that to Quibble in favor of using MediaWiki system to simulate a secondary database using a prefix or cloning the DB.

@aaron, do you think this might be attainable in CI?

I think so. It might depend on the situation. I'm thinking of the types of scenarios here:

  • Connecting to a foreign wiki domain: we can make the domains differ only by table prefix and clone those ones too (e.g. 'awiki_revision' cloned to 'unittest_awiki_revision').
  • Connecting to a virtual domain for global data: we can make the virtual domain point to the main db with a different prefix and built of the logic above.

The difficulty is that MediaWikiIntegrationTestCase would have to invoke CloneDatabase on the remote domains (e.g. 'awiki'/'bwiki'/'global'...maybe they could be hardcoded), which means that listOriginalTables() would have to be invoked a few times and support some kind of prefix parameter. All the ChangedTablesTracker stuff would also have to be checked for all the remote domains to reset tables.

Change #1309233 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):

[mediawiki/core@master] tests: Support a foreign wiki database in integration tests

https://gerrit.wikimedia.org/r/1309233