Page MenuHomePhabricator
Create Task
Maniphest T200471

LBFactorySimple breaks ExternalStorage, trying to connect to external server with local database name
Closed, ResolvedPublic
Actions

Description

Scenario:

I have a setup of 2 database servers:

  • mariadb-mw (contains main MediaWiki database, named wikidb, user wikidb)
  • mariadb-extst (contains external storage database, named wikiextst, user wikiextst)

Settings:

## Database settings
$wgDBtype = "mysql";
$wgDBserver = "localhost:/run/mysqld/mariadb-mw.sock";
$wgDBname = "wikidb";
$wgDBuser = "wikidb";
$wgDBpassword = "wikidb";

$wgDBservers = [
        [
                'host' => $wgDBserver,
                'user' => $wgDBuser,
                'password' => $wgDBpassword,
                'dbname' => $wgDBname,
                'type' => $wgDBtype,
                'flags' => DBO_DEFAULT,
                'load' => 1,
        ]
];

$wgExternalStores = array('DB');
$wgExternalServers = array( 'text1' => array (
  array( 'host' => 'localhost:/run/mysqld/mariadb-extst.sock', 'user' => 'wikiextst',  'password' =>'wikiextst',  'dbname' => 'wikiextst',  'type' => "mysql", 'load' => 1 ),
 ) );
$wgDefaultExternalStore = array ( 'DB://text1' );

This setup works perfectly fine on MediaWiki 1.30. However, on MediaWiki 1.31 access to the text of any revision (External Storage) results in a database exception:

Wikimedia\Rdbms\DBConnectionError from line 1004 of /mediawiki-1.31.0/includes/libs/rdbms/database/Database.php: 
Cannot access the database: Access denied for user 'wikiextst'@'localhost' to database 'wikidb' (localhost:/run/mysqld/mariadb-extst.sock)
#0 /mediawiki-1.31.0/includes/libs/rdbms/loadbalancer/LoadBalancer.php(1134): Wikimedia\Rdbms\Database->reportConnectionError(string)
#1 /mediawiki-1.31.0/includes/libs/rdbms/loadbalancer/LoadBalancer.php(749): Wikimedia\Rdbms\LoadBalancer->reportConnectionError()
#2 /mediawiki-1.31.0/includes/libs/rdbms/loadbalancer/LoadBalancer.php(831): Wikimedia\Rdbms\LoadBalancer->getConnection(integer, array, boolean, integer)
#3 /mediawiki-1.31.0/includes/externalstore/ExternalStoreDB.php(141): Wikimedia\Rdbms\LoadBalancer->getConnectionRef(integer, array, Wikimedia\Rdbms\DatabaseDomain)
#4 /mediawiki-1.31.0/includes/externalstore/ExternalStoreDB.php(207): ExternalStoreDB->getSlave(string)
#5 /mediawiki-1.31.0/includes/externalstore/ExternalStoreDB.php(48): ExternalStoreDB->fetchBlob(string, string, boolean)
#6 /mediawiki-1.31.0/includes/externalstore/ExternalStore.php(86): ExternalStoreDB->fetchFromURL(string)
#7 /mediawiki-1.31.0/includes/Storage/SqlBlobStore.php(400): ExternalStore::fetchFromURL(string, array)
#8 /mediawiki-1.31.0/includes/libs/objectcache/WANObjectCache.php(1240): MediaWiki\Storage\SqlBlobStore->MediaWiki\Storage\{closure}(boolean, integer, array, NULL)
#9 /mediawiki-1.31.0/includes/libs/objectcache/WANObjectCache.php(1114): WANObjectCache->doGetWithSetCallback(string, integer, Closure, array)
#10 /mediawiki-1.31.0/includes/Storage/SqlBlobStore.php(404): WANObjectCache->getWithSetCallback(string, integer, Closure, array)
#11 /mediawiki-1.31.0/includes/Storage/SqlBlobStore.php(349): MediaWiki\Storage\SqlBlobStore->expandBlob(string, array, string)
#12 /mediawiki-1.31.0/includes/Storage/SqlBlobStore.php(277): MediaWiki\Storage\SqlBlobStore->fetchBlob(string, integer)
#13 /mediawiki-1.31.0/includes/libs/objectcache/WANObjectCache.php(1240): MediaWiki\Storage\SqlBlobStore->MediaWiki\Storage\{closure}(boolean, integer, array, NULL)
#14 /mediawiki-1.31.0/includes/libs/objectcache/WANObjectCache.php(1114): WANObjectCache->doGetWithSetCallback(string, integer, Closure, array)
#15 /mediawiki-1.31.0/includes/Storage/SqlBlobStore.php(279): WANObjectCache->getWithSetCallback(string, integer, Closure, array)
#16 /mediawiki-1.31.0/includes/Storage/RevisionStore.php(921): MediaWiki\Storage\SqlBlobStore->getBlob(string, integer)
#17 /mediawiki-1.31.0/includes/Storage/RevisionStore.php(865): MediaWiki\Storage\RevisionStore->loadSlotContent(MediaWiki\Storage\SlotRecord, NULL, NULL, NULL, integer)
#18 [internal function]: MediaWiki\Storage\RevisionStore->MediaWiki\Storage\{closure}(MediaWiki\Storage\SlotRecord)
#19 /mediawiki-1.31.0/includes/Storage/SlotRecord.php(308): call_user_func(Closure, MediaWiki\Storage\SlotRecord)
#20 /mediawiki-1.31.0/includes/Storage/RevisionRecord.php(173): MediaWiki\Storage\SlotRecord->getContent()
#21 /mediawiki-1.31.0/includes/Revision.php(937): MediaWiki\Storage\RevisionRecord->getContent(string, integer, User)
#22 /mediawiki-1.31.0/includes/page/Article.php(368): Revision->getContent(integer, User)
#23 /mediawiki-1.31.0/includes/page/Article.php(562): Article->fetchContentObject()
#24 /mediawiki-1.31.0/includes/actions/ViewAction.php(68): Article->view()
#25 /mediawiki-1.31.0/includes/MediaWiki.php(500): ViewAction->show()
#26 /mediawiki-1.31.0/includes/MediaWiki.php(294): MediaWiki->performAction(Article, Title)
#27 /mediawiki-1.31.0/includes/MediaWiki.php(861): MediaWiki->performRequest()
#28 /mediawiki-1.31.0/includes/MediaWiki.php(524): MediaWiki->main()
#29 /mediawiki-1.31.0/index.php(42): MediaWiki->run()
#30 {main}

The important thing:

Cannot access the database: Access denied for user 'wikiextst'@'localhost' to database 'wikidb' (localhost:/run/mysqld/mariadb-extst.sock)

Note how it's trying to access the external storage database using the main database name. I don't understand why, because the correct database name is specified in $wgExternalServers

Details

Show related patches Customize query in gerrit

Related Objects

Event Timeline

Ciencia_Al_Poder created this task.Jul 26 2018, 9:38 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 26 2018, 9:38 PM
Ciencia_Al_Poder added a subscriber: aaron.Jul 26 2018, 9:45 PM

This may be caused by rMW14ee3f210782 self-merged by @aaron

Ciencia_Al_Poder mentioned this in T185114: Move ExternalStore table to main per-wiki database.Jul 26 2018, 9:55 PM
aaron added a comment.Jul 27 2018, 1:16 AM
In T200471#4454643, @Ciencia_Al_Poder wrote:

This may be caused by rMW14ee3f210782 self-merged by @aaron

What is that supposed to mean? I see a +2 from Daniel at https://gerrit.wikimedia.org/r/#/c/mediawiki/core/+/404056/ ? It was re +2'ed due to jenkins flakiness.

Ciencia_Al_Poder added a comment.Jul 27 2018, 4:06 PM

Yeah, sorry. The code is very complex and hard to test. I'll try to investigate in case I'm able to provide a patch, however feel free to submit a patch yourself if you can. This problem is blocking my ability to upgrade my wiki

gerritbot subscribed.Jul 27 2018, 6:25 PM

Change 448618 had a related patch set uploaded (by Martineznovo; owner: Martineznovo):
[mediawiki/core@master] Fix DatabaseDomain for ExternalStorage in LBFactorySimple

https://gerrit.wikimedia.org/r/448618

gerritbot added a project: Patch-For-Review.Jul 27 2018, 6:25 PM
Krinkle moved this task from Untriaged to Rdbms library on the MediaWiki-libs-Rdbms board.Jul 28 2018, 10:06 PM
Krinkle added a project: Performance-Team.Aug 7 2018, 4:19 PM
Krinkle subscribed.

Given this is a regression, and @aaron is already on the task and on the patch, tagging us for visibility.

gerritbot added a comment.Aug 13 2018, 6:50 PM

Change 452453 had a related patch set uploaded (by Aaron Schulz; owner: Aaron Schulz):
[mediawiki/core@master] Use LB server configuration to force DB domains in ExternalStorageDB

https://gerrit.wikimedia.org/r/452453

gerritbot added a comment.Aug 13 2018, 7:01 PM

Change 452455 had a related patch set uploaded (by Aaron Schulz; owner: Aaron Schulz):
[mediawiki/core@REL1_31] Use LB server configuration to force DB domains in ExternalStorageDB

https://gerrit.wikimedia.org/r/452455

• Imarlier moved this task from Inbox, needs triage to Doing (old) on the Performance-Team board.Aug 13 2018, 8:16 PM
aaron claimed this task.Aug 13 2018, 8:16 PM
gerritbot added a comment.Aug 13 2018, 10:30 PM

Change 452453 merged by jenkins-bot:
[mediawiki/core@master] Use LB server configuration to force DB domains in ExternalStorageDB

https://gerrit.wikimedia.org/r/452453

gerritbot added a comment.Aug 13 2018, 11:00 PM

Change 452455 merged by jenkins-bot:
[mediawiki/core@REL1_31] Use LB server configuration to force DB domains in ExternalStorageDB

https://gerrit.wikimedia.org/r/452455

ReleaseTaggerBot added projects: MW-1.32-notes (WMF-deploy-2018-08-14 (1.32.0-wmf.17)), MW-1.31-release-notes.Aug 13 2018, 11:00 PM
gerritbot added a comment.Aug 14 2018, 5:56 PM

Change 448618 abandoned by Martineznovo:
Fix DatabaseDomain for ExternalStorage in LBFactorySimple

Reason:
There's a better one now

https://gerrit.wikimedia.org/r/448618

Ciencia_Al_Poder closed this task as Resolved.Aug 14 2018, 5:56 PM
Ciencia_Al_Poder removed a project: Patch-For-Review.
Jdforrester-WMF edited projects, added MW-1.32-notes (WMF-deploy-2018-08-21 (1.32.0-wmf.18)); removed MW-1.32-notes (WMF-deploy-2018-08-14 (1.32.0-wmf.17)).Aug 14 2018, 8:54 PM
gerritbot added a comment.Aug 15 2018, 3:03 AM

Change 452880 had a related patch set uploaded (by Krinkle; owner: Aaron Schulz):
[mediawiki/core@master] Make ExternalStoreDB "wiki" context override the server "dbname" field

https://gerrit.wikimedia.org/r/452880

gerritbot added a project: Patch-For-Review.Aug 15 2018, 3:03 AM

Change 452881 had a related patch set uploaded (by Krinkle; owner: Aaron Schulz):
[mediawiki/core@REL1_31] Make ExternalStoreDB "wiki" context override the server "dbname" field

https://gerrit.wikimedia.org/r/452881

gerritbot added a comment.Aug 15 2018, 5:34 AM

Change 452880 merged by jenkins-bot:
[mediawiki/core@master] Make ExternalStoreDB "wiki" context override the server "dbname" field

https://gerrit.wikimedia.org/r/452880

gerritbot added a comment.Aug 21 2018, 1:33 AM

Change 452881 merged by jenkins-bot:
[mediawiki/core@REL1_31] Make ExternalStoreDB "wiki" context override the server "dbname" field

https://gerrit.wikimedia.org/r/452881

Ciencia_Al_Poder reopened this task as Open.Apr 20 2019, 9:34 PM

I'm sorry this is still happening

When I commented on https://gerrit.wikimedia.org/r/452453/ I tested with that patch only and it worked, that's why I marked it as RESOLVED.

However, later @aaron authored https://gerrit.wikimedia.org/r/452880 which breaks it again, in MediaWiki 1.31, 1.32 and current master.

I'm sorry I couldn't test this new release until now, I had some problems that prevented me from having time to upgrade and retest everything.

What is the purpose of that new patchset and what it's supposed to fix?

Krinkle added a comment.Apr 21 2019, 3:35 PM

@Ciencia_Al_Poder From a quick glance, the referenced patch adds a condition for params['wiki'], however I do not see this being set in your configuration. If the configuration has changed or wasn't as described, please provide an updated overview of your DB and ES-related settings.

Ciencia_Al_Poder added a comment.Apr 21 2019, 3:53 PM

Hello Krinkle. I was trying to hunt the bug without noticing the latest patch from aaron after the bug was marked as resolved, so I did test with a clean empty wiki downloaded from git master and only this addition to the default LocalSettings.php that the installer generates:

$wgExternalStores = array('DB');
$wgExternalServers = array( 'text1' => array (
  array( 'host' => 'localhost:/run/mysql/mysql.sock', 'user' => 'mwgitextst',  'password' =>'mwgitextst',  'dbname' => 'mwgitextst',  'type' => "mysql", 'load' => 1 ),
 ) );
$wgDefaultExternalStore = array ( 'DB://text1' );
$wgCompressRevisions = true;

And then created the new database running this SQL:

create database mwgitextst;

CREATE USER 'mwgitextst'@'localhost' IDENTIFIED BY 'mwgitextst' REQUIRE NONE;
GRANT SELECT,INSERT,UPDATE,DELETE ON mwgitextst . * TO 'mwgitextst'@'localhost';

CREATE TABLE blobs (
	blob_id integer UNSIGNED NOT NULL AUTO_INCREMENT,
	blob_text longblob,
	PRIMARY KEY  (blob_id)
) ENGINE=InnoDB;

With this configuration, this is what happens:

  1. Open a new page for editing.
  2. Write something and hit save changes
  3. The page is saved and the page is reloaded in view mode (everything is working perfectly until here)
  4. Problem: When I hit "edit" (on the page I've just created), I get a Wikimedia\Rdbms\LoadBalancer->reportConnectionError().

real_connect(): (HY000/1044): Access denied for user 'mwgitextst'@'localhost' to database 'mwgit'

It's trying to retrieve the external storage data from the local database instead of the foreign one, using the foreign database credentials.

I'm trying to figure out the purpose of the "wiki" parameter, but the code is a real mess, assuming in a lot of places that it wants to use the "local" database when we're trying to get the "external" storage.

And the most surprising thing here is that it works when saving the edit, but not when retrieving it! The database for external storage contains the new text.

I'm trying to see if there's another solution instead of just reverting that patch, but if you see it clearer where's the problem, feel free to upload a patch and I'm more than happy to test that.

Krinkle renamed this task from [regression] LBFactorySimple breaks ExternalStorage, trying to connect to external server with local database name to LBFactorySimple breaks ExternalStorage, trying to connect to external server with local database name.Apr 21 2019, 4:22 PM
Krinkle edited projects, added MW-1.33-release; removed MW-1.32-notes (WMF-deploy-2018-08-21 (1.32.0-wmf.18)), MW-1.31-release-notes.
Krinkle removed a project: Patch-For-Review.
gerritbot added a comment.Apr 21 2019, 7:57 PM

Change 505468 had a related patch set uploaded (by Aaron Schulz; owner: Aaron Schulz):
[mediawiki/core@master] externalstore: make ExternalStoreDB::getDomainId treat false the same as null

https://gerrit.wikimedia.org/r/505468

gerritbot added a project: Patch-For-Review.Apr 21 2019, 7:57 PM
gerritbot added a comment.Apr 22 2019, 9:27 AM

Change 505625 had a related patch set uploaded (by Martineznovo; owner: Martineznovo):
[mediawiki/core@master] ExternalStore: Pass external domain to getReadOnlyReason

https://gerrit.wikimedia.org/r/505625

gerritbot added a comment.May 22 2019, 10:38 AM

Change 505468 merged by jenkins-bot:
[mediawiki/core@master] externalstore: make ExternalStoreDB::getDomainId treat false the same as null

https://gerrit.wikimedia.org/r/505468

gerritbot added a comment.May 22 2019, 7:26 PM

Change 505625 merged by jenkins-bot:
[mediawiki/core@master] ExternalStore: Pass external domain to getReadOnlyReason

https://gerrit.wikimedia.org/r/505625

gerritbot added a comment.May 22 2019, 8:18 PM

Change 511980 had a related patch set uploaded (by Martineznovo; owner: Aaron Schulz):
[mediawiki/core@REL1_33] externalstore: make ExternalStoreDB::getDomainId treat false the same as null

https://gerrit.wikimedia.org/r/511980

gerritbot added a comment.May 22 2019, 8:19 PM

Change 511981 had a related patch set uploaded (by Martineznovo; owner: Martineznovo):
[mediawiki/core@REL1_33] ExternalStore: Pass external domain to getReadOnlyReason

https://gerrit.wikimedia.org/r/511981

ReleaseTaggerBot added a project: MW-1.34-notes (1.34.0-wmf.7; 2019-05-28).May 25 2019, 1:01 AM
gerritbot added a comment.Jun 1 2019, 12:46 AM

Change 511980 merged by jenkins-bot:
[mediawiki/core@REL1_33] externalstore: make ExternalStoreDB::getDomainId treat false the same as null

https://gerrit.wikimedia.org/r/511980

gerritbot added a comment.Jun 1 2019, 12:46 AM

Change 511981 merged by jenkins-bot:
[mediawiki/core@REL1_33] ExternalStore: Pass external domain to getReadOnlyReason

https://gerrit.wikimedia.org/r/511981

ReleaseTaggerBot added a project: MW-1.33-notes.Jun 1 2019, 1:00 AM
aaron added a comment.Jun 1 2019, 8:33 AM

What is the status of this after the above patches are applied?

Ciencia_Al_Poder added a comment.Jun 1 2019, 3:55 PM

Now External Storage works on REL1_33 (tested it), and on REL1_32 if I manually apply the patches. May be worth backporting (they're applied and work well on a production wiki on 1.32)

gerritbot added a comment.Jun 1 2019, 5:20 PM

Change 513734 had a related patch set uploaded (by Krinkle; owner: Martineznovo):
[mediawiki/core@REL1_32] ExternalStore: Pass external domain to getReadOnlyReason

https://gerrit.wikimedia.org/r/513734

gerritbot added a comment.Jun 1 2019, 5:21 PM

Change 513735 had a related patch set uploaded (by Krinkle; owner: Aaron Schulz):
[mediawiki/core@REL1_32] externalstore: make ExternalStoreDB::getDomainId treat false the same as null

https://gerrit.wikimedia.org/r/513735

Krinkle closed this task as Resolved.Jun 1 2019, 5:22 PM
Krinkle removed projects: MW-1.34-notes (1.34.0-wmf.7; 2019-05-28), Patch-For-Review, MW-1.33-release.
gerritbot added a comment.Jun 1 2019, 5:30 PM

Change 513735 merged by jenkins-bot:
[mediawiki/core@REL1_32] externalstore: make ExternalStoreDB::getDomainId treat false the same as null

https://gerrit.wikimedia.org/r/513735

gerritbot added a comment.Jun 1 2019, 5:30 PM

Change 513734 merged by jenkins-bot:
[mediawiki/core@REL1_32] ExternalStore: Pass external domain to getReadOnlyReason

https://gerrit.wikimedia.org/r/513734

ReleaseTaggerBot added a project: MW-1.32-notes.Jun 1 2019, 10:00 PM
Anomie mentioned this in T212881: addWiki.php broken creating ES tables.Sep 5 2019, 2:07 PM
TK-999 mentioned this in T236374: ExternalStoreDB fails to connect for foreign wikis when custom external storage database names are used.Oct 24 2019, 12:19 PM
daniel mentioned this in T275452: addWiki.php throws Wikimedia\Assert\PreconditionException from line 58 of /srv/mediawiki/php-1.36.0-wmf.31/includes/dao/WikiAwareEntityTrait.php: Expected User to belong to 'altwiki', but it belongs to the local wiki on writing main page.Feb 23 2021, 3:04 PM
gerritbot added a comment.Dec 15 2021, 2:32 AM

Change 747225 had a related patch set uploaded (by Krinkle; author: Krinkle):

[mediawiki/core@master] externalstore: Always use 'dbname' if set for an external cluster

https://gerrit.wikimedia.org/r/747225

gerritbot added a project: Patch-For-Review.Dec 15 2021, 2:32 AM
gerritbot added a comment.May 2 2022, 6:01 PM

Change 747225 abandoned by Krinkle:

[mediawiki/core@master] externalstore: Always use 'dbname' if set for an external cluster

Reason:

https://gerrit.wikimedia.org/r/747225

Umherirrender mentioned this in T304528: PHP Deprecated: Deprecated cross-wiki access to MediaWiki\Revision\RevisionRecord. Expected: the local wiki, Actual: 'guwwiki'. Pass expected $wikiId. [Called from MediaWiki\Revision\RevisionRecord::getPageId].Aug 4 2022, 10:16 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL