Page MenuHomePhabricator

LBFactorySimple breaks ExternalStorage, trying to connect to external server with local database name
Open, Needs TriagePublic

Description

Scenario:

I have a setup of 2 database servers:

  • mariadb-mw (contains main MediaWiki database, named wikidb, user wikidb)
  • mariadb-extst (contains external storage database, named wikiextst, user wikiextst)

Settings:

## Database settings
$wgDBtype = "mysql";
$wgDBserver = "localhost:/run/mysqld/mariadb-mw.sock";
$wgDBname = "wikidb";
$wgDBuser = "wikidb";
$wgDBpassword = "wikidb";

$wgDBservers = [
        [
                'host' => $wgDBserver,
                'user' => $wgDBuser,
                'password' => $wgDBpassword,
                'dbname' => $wgDBname,
                'type' => $wgDBtype,
                'flags' => DBO_DEFAULT,
                'load' => 1,
        ]
];

$wgExternalStores = array('DB');
$wgExternalServers = array( 'text1' => array (
  array( 'host' => 'localhost:/run/mysqld/mariadb-extst.sock', 'user' => 'wikiextst',  'password' =>'wikiextst',  'dbname' => 'wikiextst',  'type' => "mysql", 'load' => 1 ),
 ) );
$wgDefaultExternalStore = array ( 'DB://text1' );

This setup works perfectly fine on MediaWiki 1.30. However, on MediaWiki 1.31 access to the text of any revision (External Storage) results in a database exception:

Wikimedia\Rdbms\DBConnectionError from line 1004 of /mediawiki-1.31.0/includes/libs/rdbms/database/Database.php: 
Cannot access the database: Access denied for user 'wikiextst'@'localhost' to database 'wikidb' (localhost:/run/mysqld/mariadb-extst.sock)
#0 /mediawiki-1.31.0/includes/libs/rdbms/loadbalancer/LoadBalancer.php(1134): Wikimedia\Rdbms\Database->reportConnectionError(string)
#1 /mediawiki-1.31.0/includes/libs/rdbms/loadbalancer/LoadBalancer.php(749): Wikimedia\Rdbms\LoadBalancer->reportConnectionError()
#2 /mediawiki-1.31.0/includes/libs/rdbms/loadbalancer/LoadBalancer.php(831): Wikimedia\Rdbms\LoadBalancer->getConnection(integer, array, boolean, integer)
#3 /mediawiki-1.31.0/includes/externalstore/ExternalStoreDB.php(141): Wikimedia\Rdbms\LoadBalancer->getConnectionRef(integer, array, Wikimedia\Rdbms\DatabaseDomain)
#4 /mediawiki-1.31.0/includes/externalstore/ExternalStoreDB.php(207): ExternalStoreDB->getSlave(string)
#5 /mediawiki-1.31.0/includes/externalstore/ExternalStoreDB.php(48): ExternalStoreDB->fetchBlob(string, string, boolean)
#6 /mediawiki-1.31.0/includes/externalstore/ExternalStore.php(86): ExternalStoreDB->fetchFromURL(string)
#7 /mediawiki-1.31.0/includes/Storage/SqlBlobStore.php(400): ExternalStore::fetchFromURL(string, array)
#8 /mediawiki-1.31.0/includes/libs/objectcache/WANObjectCache.php(1240): MediaWiki\Storage\SqlBlobStore->MediaWiki\Storage\{closure}(boolean, integer, array, NULL)
#9 /mediawiki-1.31.0/includes/libs/objectcache/WANObjectCache.php(1114): WANObjectCache->doGetWithSetCallback(string, integer, Closure, array)
#10 /mediawiki-1.31.0/includes/Storage/SqlBlobStore.php(404): WANObjectCache->getWithSetCallback(string, integer, Closure, array)
#11 /mediawiki-1.31.0/includes/Storage/SqlBlobStore.php(349): MediaWiki\Storage\SqlBlobStore->expandBlob(string, array, string)
#12 /mediawiki-1.31.0/includes/Storage/SqlBlobStore.php(277): MediaWiki\Storage\SqlBlobStore->fetchBlob(string, integer)
#13 /mediawiki-1.31.0/includes/libs/objectcache/WANObjectCache.php(1240): MediaWiki\Storage\SqlBlobStore->MediaWiki\Storage\{closure}(boolean, integer, array, NULL)
#14 /mediawiki-1.31.0/includes/libs/objectcache/WANObjectCache.php(1114): WANObjectCache->doGetWithSetCallback(string, integer, Closure, array)
#15 /mediawiki-1.31.0/includes/Storage/SqlBlobStore.php(279): WANObjectCache->getWithSetCallback(string, integer, Closure, array)
#16 /mediawiki-1.31.0/includes/Storage/RevisionStore.php(921): MediaWiki\Storage\SqlBlobStore->getBlob(string, integer)
#17 /mediawiki-1.31.0/includes/Storage/RevisionStore.php(865): MediaWiki\Storage\RevisionStore->loadSlotContent(MediaWiki\Storage\SlotRecord, NULL, NULL, NULL, integer)
#18 [internal function]: MediaWiki\Storage\RevisionStore->MediaWiki\Storage\{closure}(MediaWiki\Storage\SlotRecord)
#19 /mediawiki-1.31.0/includes/Storage/SlotRecord.php(308): call_user_func(Closure, MediaWiki\Storage\SlotRecord)
#20 /mediawiki-1.31.0/includes/Storage/RevisionRecord.php(173): MediaWiki\Storage\SlotRecord->getContent()
#21 /mediawiki-1.31.0/includes/Revision.php(937): MediaWiki\Storage\RevisionRecord->getContent(string, integer, User)
#22 /mediawiki-1.31.0/includes/page/Article.php(368): Revision->getContent(integer, User)
#23 /mediawiki-1.31.0/includes/page/Article.php(562): Article->fetchContentObject()
#24 /mediawiki-1.31.0/includes/actions/ViewAction.php(68): Article->view()
#25 /mediawiki-1.31.0/includes/MediaWiki.php(500): ViewAction->show()
#26 /mediawiki-1.31.0/includes/MediaWiki.php(294): MediaWiki->performAction(Article, Title)
#27 /mediawiki-1.31.0/includes/MediaWiki.php(861): MediaWiki->performRequest()
#28 /mediawiki-1.31.0/includes/MediaWiki.php(524): MediaWiki->main()
#29 /mediawiki-1.31.0/index.php(42): MediaWiki->run()
#30 {main}

The important thing:

Cannot access the database: Access denied for user 'wikiextst'@'localhost' to database 'wikidb' (localhost:/run/mysqld/mariadb-extst.sock)

Note how it's trying to access the external storage database using the main database name. I don't understand why, because the correct database name is specified in $wgExternalServers

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 26 2018, 9:38 PM

This may be caused by rMW14ee3f210782 self-merged by @aaron

aaron added a comment.Jul 27 2018, 1:16 AM

This may be caused by rMW14ee3f210782 self-merged by @aaron

What is that supposed to mean? I see a +2 from Daniel at https://gerrit.wikimedia.org/r/#/c/mediawiki/core/+/404056/ ? It was re +2'ed due to jenkins flakiness.

Yeah, sorry. The code is very complex and hard to test. I'll try to investigate in case I'm able to provide a patch, however feel free to submit a patch yourself if you can. This problem is blocking my ability to upgrade my wiki

Change 448618 had a related patch set uploaded (by Martineznovo; owner: Martineznovo):
[mediawiki/core@master] Fix DatabaseDomain for ExternalStorage in LBFactorySimple

https://gerrit.wikimedia.org/r/448618

Krinkle added a subscriber: Krinkle.

Given this is a regression, and @aaron is already on the task and on the patch, tagging us for visibility.

Change 452453 had a related patch set uploaded (by Aaron Schulz; owner: Aaron Schulz):
[mediawiki/core@master] Use LB server configuration to force DB domains in ExternalStorageDB

https://gerrit.wikimedia.org/r/452453

Change 452455 had a related patch set uploaded (by Aaron Schulz; owner: Aaron Schulz):
[mediawiki/core@REL1_31] Use LB server configuration to force DB domains in ExternalStorageDB

https://gerrit.wikimedia.org/r/452455

Imarlier moved this task from Inbox to Doing on the Performance-Team board.Aug 13 2018, 8:16 PM
aaron claimed this task.Aug 13 2018, 8:16 PM

Change 452453 merged by jenkins-bot:
[mediawiki/core@master] Use LB server configuration to force DB domains in ExternalStorageDB

https://gerrit.wikimedia.org/r/452453

Change 452455 merged by jenkins-bot:
[mediawiki/core@REL1_31] Use LB server configuration to force DB domains in ExternalStorageDB

https://gerrit.wikimedia.org/r/452455

Change 448618 abandoned by Martineznovo:
Fix DatabaseDomain for ExternalStorage in LBFactorySimple

Reason:
There's a better one now

https://gerrit.wikimedia.org/r/448618

Ciencia_Al_Poder closed this task as Resolved.Aug 14 2018, 5:56 PM

Change 452880 had a related patch set uploaded (by Krinkle; owner: Aaron Schulz):
[mediawiki/core@master] Make ExternalStoreDB "wiki" context override the server "dbname" field

https://gerrit.wikimedia.org/r/452880

Change 452881 had a related patch set uploaded (by Krinkle; owner: Aaron Schulz):
[mediawiki/core@REL1_31] Make ExternalStoreDB "wiki" context override the server "dbname" field

https://gerrit.wikimedia.org/r/452881

Change 452880 merged by jenkins-bot:
[mediawiki/core@master] Make ExternalStoreDB "wiki" context override the server "dbname" field

https://gerrit.wikimedia.org/r/452880

Change 452881 merged by jenkins-bot:
[mediawiki/core@REL1_31] Make ExternalStoreDB "wiki" context override the server "dbname" field

https://gerrit.wikimedia.org/r/452881

Ciencia_Al_Poder reopened this task as Open.Sat, Apr 20, 9:34 PM

I'm sorry this is still happening

When I commented on https://gerrit.wikimedia.org/r/452453/ I tested with that patch only and it worked, that's why I marked it as RESOLVED.

However, later @aaron authored https://gerrit.wikimedia.org/r/452880 which breaks it again, in MediaWiki 1.31, 1.32 and current master.

I'm sorry I couldn't test this new release until now, I had some problems that prevented me from having time to upgrade and retest everything.

What is the purpose of that new patchset and what it's supposed to fix?

@Ciencia_Al_Poder From a quick glance, the referenced patch adds a condition for params['wiki'], however I do not see this being set in your configuration. If the configuration has changed or wasn't as described, please provide an updated overview of your DB and ES-related settings.

Hello Krinkle. I was trying to hunt the bug without noticing the latest patch from aaron after the bug was marked as resolved, so I did test with a clean empty wiki downloaded from git master and only this addition to the default LocalSettings.php that the installer generates:

$wgExternalStores = array('DB');
$wgExternalServers = array( 'text1' => array (
  array( 'host' => 'localhost:/run/mysql/mysql.sock', 'user' => 'mwgitextst',  'password' =>'mwgitextst',  'dbname' => 'mwgitextst',  'type' => "mysql", 'load' => 1 ),
 ) );
$wgDefaultExternalStore = array ( 'DB://text1' );
$wgCompressRevisions = true;

And then created the new database running this SQL:

create database mwgitextst;

CREATE USER 'mwgitextst'@'localhost' IDENTIFIED BY 'mwgitextst' REQUIRE NONE;
GRANT SELECT,INSERT,UPDATE,DELETE ON mwgitextst . * TO 'mwgitextst'@'localhost';

CREATE TABLE blobs (
	blob_id integer UNSIGNED NOT NULL AUTO_INCREMENT,
	blob_text longblob,
	PRIMARY KEY  (blob_id)
) ENGINE=InnoDB;

With this configuration, this is what happens:

  1. Open a new page for editing.
  2. Write something and hit save changes
  3. The page is saved and the page is reloaded in view mode (everything is working perfectly until here)
  4. Problem: When I hit "edit" (on the page I've just created), I get a Wikimedia\Rdbms\LoadBalancer->reportConnectionError().

real_connect(): (HY000/1044): Access denied for user 'mwgitextst'@'localhost' to database 'mwgit'

It's trying to retrieve the external storage data from the local database instead of the foreign one, using the foreign database credentials.

I'm trying to figure out the purpose of the "wiki" parameter, but the code is a real mess, assuming in a lot of places that it wants to use the "local" database when we're trying to get the "external" storage.

And the most surprising thing here is that it works when saving the edit, but not when retrieving it! The database for external storage contains the new text.

I'm trying to see if there's another solution instead of just reverting that patch, but if you see it clearer where's the problem, feel free to upload a patch and I'm more than happy to test that.

Krinkle removed a project: Patch-For-Review.
Krinkle renamed this task from [regression] LBFactorySimple breaks ExternalStorage, trying to connect to external server with local database name to LBFactorySimple breaks ExternalStorage, trying to connect to external server with local database name.

Change 505468 had a related patch set uploaded (by Aaron Schulz; owner: Aaron Schulz):
[mediawiki/core@master] externalstore: make ExternalStoreDB::getDomainId treat false the same as null

https://gerrit.wikimedia.org/r/505468

Change 505625 had a related patch set uploaded (by Martineznovo; owner: Martineznovo):
[mediawiki/core@master] ExternalStore: Pass external domain to getReadOnlyReason

https://gerrit.wikimedia.org/r/505625