Page MenuHomePhabricator

Wikibase dump scripts fail on external storage access
Closed, ResolvedPublic

Description

Wikimedia\Rdbms\DBConnectionError from line 1392 of /srv/mediawiki/php-1.34.0-wmf.13/includes/libs/rdbms/loadbalancer/LoadBalancer.php: Cannot access the database: No working replica DB server: Unknown error
#0 /srv/mediawiki/php-1.34.0-wmf.13/includes/libs/rdbms/loadbalancer/LoadBalancer.php(467): Wikimedia\Rdbms\LoadBalancer->reportConnectionError()
#1 /srv/mediawiki/php-1.34.0-wmf.13/includes/libs/rdbms/loadbalancer/LoadBalancer.php(897): Wikimedia\Rdbms\LoadBalancer->getConnectionIndex(false, Array, 'wikidatawiki')
#2 /srv/mediawiki/php-1.34.0-wmf.13/includes/libs/rdbms/loadbalancer/LoadBalancer.php(1035): Wikimedia\Rdbms\LoadBalancer->getConnection(-1, Array, 'wikidatawiki', 0)
#3 /srv/mediawiki/php-1.34.0-wmf.13/includes/externalstore/ExternalStoreDB.php(166): Wikimedia\Rdbms\LoadBalancer->getConnectionRef(-1, Array, 'wikidatawiki')
#4 /srv/mediawiki/php-1.34.0-wmf.13/includes/externalstore/ExternalStoreDB.php(259): ExternalStoreDB->getSlave('cluster25')
#5 /srv/mediawiki/php-1.34.0-wmf.13/includes/externalstore/ExternalStoreDB.php(65): ExternalStoreDB->fetchBlob('cluster25', '487121788', false)
#6 /srv/mediawiki/php-1.34.0-wmf.13/includes/externalstore/ExternalStoreAccess.php(52): ExternalStoreDB->fetchFromURL('DB://cluster25/...')
#7 /srv/mediawiki/php-1.34.0-wmf.13/includes/Storage/SqlBlobStore.php(427): ExternalStoreAccess->fetchFromURL('DB://cluster25/...', Array)
#8 /srv/mediawiki/php-1.34.0-wmf.13/includes/libs/objectcache/WANObjectCache.php(1406): MediaWiki\Storage\SqlBlobStore->MediaWiki\Storage\{closure}(false, 604800, Array, NULL)
#9 /srv/mediawiki/php-1.34.0-wmf.13/includes/libs/objectcache/WANObjectCache.php(1269): WANObjectCache->doGetWithSetCallback('global:BlobStor...', 604800, Object(Closure), Array)
#10 /srv/mediawiki/php-1.34.0-wmf.13/includes/Storage/SqlBlobStore.php(431): WANObjectCache->getWithSetCallback('global:BlobStor...', 604800, Object(Closure), Array)
#11 /srv/mediawiki/php-1.34.0-wmf.13/includes/Storage/SqlBlobStore.php(358): MediaWiki\Storage\SqlBlobStore->expandBlob('DB://cluster25/...', Array, 'tt:971848740')
#12 /srv/mediawiki/php-1.34.0-wmf.13/includes/Storage/SqlBlobStore.php(286): MediaWiki\Storage\SqlBlobStore->fetchBlob('tt:971848740', 0)
#13 /srv/mediawiki/php-1.34.0-wmf.13/includes/libs/objectcache/WANObjectCache.php(1406): MediaWiki\Storage\SqlBlobStore->MediaWiki\Storage\{closure}(false, 604800, Array, NULL)
#14 /srv/mediawiki/php-1.34.0-wmf.13/includes/libs/objectcache/WANObjectCache.php(1269): WANObjectCache->doGetWithSetCallback('global:BlobStor...', 604800, Object(Closure), Array)
#15 /srv/mediawiki/php-1.34.0-wmf.13/includes/Storage/SqlBlobStore.php(288): WANObjectCache->getWithSetCallback('global:BlobStor...', 604800, Object(Closure), Array)
#16 /srv/mediawiki/php-1.34.0-wmf.13/extensions/Wikibase/lib/includes/Store/Sql/WikiPageEntityRevisionLookup.php(241): MediaWiki\Storage\SqlBlobStore->getBlob('tt:971848740', 0)
#17 /srv/mediawiki/php-1.34.0-wmf.13/extensions/Wikibase/lib/includes/Store/Sql/WikiPageEntityRevisionLookup.php(132): Wikibase\Lib\Store\Sql\WikiPageEntityRevisionLookup->loadEntity(Object(stdClass), 'replica')
#18 /srv/mediawiki/php-1.34.0-wmf.13/extensions/Wikibase/lib/includes/Store/DispatchingEntityRevisionLookup.php(59): Wikibase\Lib\Store\Sql\WikiPageEntityRevisionLookup->getEntityRevision(Object(Wikibase\DataModel\Entity\ItemId), 0, 'replica')
#19 /srv/mediawiki/php-1.34.0-wmf.13/extensions/Wikibase/lib/includes/Store/TypeDispatchingEntityRevisionLookup.php(54): Wikibase\Lib\Store\DispatchingEntityRevisionLookup->getEntityRevision(Object(Wikibase\DataModel\Entity\ItemId), 0, 'replica')
#20 /srv/mediawiki/php-1.34.0-wmf.13/extensions/Wikibase/lib/includes/Store/CacheRetrievingEntityRevisionLookup.php(75): Wikibase\Lib\Store\TypeDispatchingEntityRevisionLookup->getEntityRevision(Object(Wikibase\DataModel\Entity\ItemId), 0, 'replica')
#21 /srv/mediawiki/php-1.34.0-wmf.13/extensions/Wikibase/repo/includes/Dumpers/JsonDumpGenerator.php(128): Wikibase\Lib\Store\CacheRetrievingEntityRevisionLookup->getEntityRevision(Object(Wikibase\DataModel\Entity\ItemId))
#22 /srv/mediawiki/php-1.34.0-wmf.13/extensions/Wikibase/repo/includes/Dumpers/DumpGenerator.php(299): Wikibase\Dumpers\JsonDumpGenerator->generateDumpForEntityId(Object(Wikibase\DataModel\Entity\ItemId))
#23 /srv/mediawiki/php-1.34.0-wmf.13/extensions/Wikibase/repo/includes/Dumpers/DumpGenerator.php(262): Wikibase\Dumpers\DumpGenerator->dumpEntities(Array, 879)
#24 /srv/mediawiki/php-1.34.0-wmf.13/extensions/Wikibase/repo/maintenance/DumpEntities.php(221): Wikibase\Dumpers\DumpGenerator->generateDump(Object(Wikibase\Repo\Store\Sql\SqlEntityIdPager))
#25 /srv/mediawiki/php-1.34.0-wmf.13/extensions/Wikibase/repo/maintenance/dumpJson.php(100): Wikibase\DumpEntities->execute()
#26 /srv/mediawiki/php-1.34.0-wmf.13/maintenance/doMaintenance.php(99): Wikibase\DumpJson->execute()
#27 /srv/mediawiki/php-1.34.0-wmf.13/extensions/Wikibase/repo/maintenance/dumpJson.php(126): require_once('/srv/mediawiki/...')
#28 /srv/mediawiki/multiversion/MWScript.php(101): require_once('/srv/mediawiki/...')
#29 {main}

Event Timeline

hoo created this task.Jul 15 2019, 9:01 PM
Restricted Application added a project: Wikidata. · View Herald TranscriptJul 15 2019, 9:01 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
hoo added a comment.Jul 15 2019, 9:06 PM

This is again related to dbgroupdefault (caused by ab09cf2fa59).

hoo@mwmaint1002:~$ mwscript eval.php --wiki wikidatawiki
> $mediaWikiServices = MediaWiki\MediaWikiServices::getInstance(); $exStore = $mediaWikiServices->getExternalStoreAccess(); $exStore->fetchFromURL('DB://cluster25/487121788');

> ^D
hoo@mwmaint1002:~$ mwscript eval.php --wiki wikidatawiki --dbgroupdefault dumps
> $mediaWikiServices = MediaWiki\MediaWikiServices::getInstance(); $exStore = $mediaWikiServices->getExternalStoreAccess(); $exStore->fetchFromURL('DB://cluster25/487121788');
Caught exception Wikimedia\Rdbms\DBConnectionError: Cannot access the database: No working replica DB server: Unknown error
#0 /srv/mediawiki/php-1.34.0-wmf.13/includes/libs/rdbms/loadbalancer/LoadBalancer.php(467): Wikimedia\Rdbms\LoadBalancer->reportConnectionError()
#1 /srv/mediawiki/php-1.34.0-wmf.13/includes/libs/rdbms/loadbalancer/LoadBalancer.php(897): Wikimedia\Rdbms\LoadBalancer->getConnectionIndex()
#2 /srv/mediawiki/php-1.34.0-wmf.13/includes/libs/rdbms/loadbalancer/LoadBalancer.php(1035): Wikimedia\Rdbms\LoadBalancer->getConnection()
#3 /srv/mediawiki/php-1.34.0-wmf.13/includes/externalstore/ExternalStoreDB.php(166): Wikimedia\Rdbms\LoadBalancer->getConnectionRef()
#4 /srv/mediawiki/php-1.34.0-wmf.13/includes/externalstore/ExternalStoreDB.php(259): ExternalStoreDB->getSlave()
#5 /srv/mediawiki/php-1.34.0-wmf.13/includes/externalstore/ExternalStoreDB.php(65): ExternalStoreDB->fetchBlob()
#6 /srv/mediawiki/php-1.34.0-wmf.13/includes/externalstore/ExternalStoreAccess.php(52): ExternalStoreDB->fetchFromURL()
#7 /srv/mediawiki/php-1.34.0-wmf.13/maintenance/eval.php(78)(57f4d4e974a5dc7047f3cca4b7fdb441) : eval()'d code(1): ExternalStoreAccess->fetchFromURL()
#8 /srv/mediawiki/php-1.34.0-wmf.13/maintenance/eval.php(78): include()
#9 /srv/mediawiki/multiversion/MWScript.php(101): include()
#10 {main}
>

Change 521920 had a related patch set uploaded (by Hoo man; owner: Hoo man):
[mediawiki/core@master] LoadBalancer::getConnectionIndex: Also fallback to generic group

https://gerrit.wikimedia.org/r/521920

Change 521920 abandoned by Hoo man:
LoadBalancer::getConnectionIndex: Also fallback to generic group

Reason:
Better solved through https://gerrit.wikimedia.org/r/c/mediawiki/core/ /523412

https://gerrit.wikimedia.org/r/521920

Change 523990 had a related patch set uploaded (by Jforrester; owner: Aaron Schulz):
[mediawiki/core@wmf/1.34.0-wmf.14] rdbms: better handle a non-existing "defaultGroup" in LoadBalancer

https://gerrit.wikimedia.org/r/523990

Change 523990 merged by jenkins-bot:
[mediawiki/core@wmf/1.34.0-wmf.14] rdbms: better handle a non-existing "defaultGroup" in LoadBalancer

https://gerrit.wikimedia.org/r/523990

Mentioned in SAL (#wikimedia-operations) [2019-07-17T19:35:56Z] <jforrester@deploy1001> Synchronized php-1.34.0-wmf.14/includes/libs/rdbms/loadbalancer: T228104 rdbms: better handle a non-existing defaultGroup in LoadBalancer (duration: 00m 55s)

Change 523997 had a related patch set uploaded (by Jforrester; owner: Aaron Schulz):
[mediawiki/core@wmf/1.34.0-wmf.13] rdbms: better handle a non-existing "defaultGroup" in LoadBalancer

https://gerrit.wikimedia.org/r/523997

hoo added a comment.Jul 17 2019, 7:42 PM

We backported this to wmf14 now to check for fallout. If there's none, we will also target wmf13 later on.

Change 523997 merged by jenkins-bot:
[mediawiki/core@wmf/1.34.0-wmf.13] rdbms: better handle a non-existing "defaultGroup" in LoadBalancer

https://gerrit.wikimedia.org/r/523997

Mentioned in SAL (#wikimedia-operations) [2019-07-17T21:16:52Z] <jforrester@deploy1001> Synchronized php-1.34.0-wmf.13/includes/libs/rdbms/loadbalancer: T228104 rdbms: better handle a non-existing defaultGroup in LoadBalancer (duration: 00m 55s)

This is now deployed to wmf.13. Can we trigger a dumps run to show that it works?

I have started the json format dumps and they seem to be running correctly.

Envlh added a subscriber: Envlh.Jul 17 2019, 9:57 PM

Mentioned in SAL (#wikimedia-operations) [2019-07-17T22:16:22Z] <hoo> Manually started the Wikidata RDF dumps on snapshot1008 (due to T228104)

ArielGlenn closed this task as Resolved.Jul 19 2019, 7:36 AM
ArielGlenn claimed this task.

The above changes were deployed everywhere on Wednesday. JSON and Rdf dumps have been running since then without issues. The JSON dumps should complete sometime today. Closing this, whew!

Envlh added a comment.Jul 19 2019, 4:56 PM

Thank you all! :)

ArielGlenn moved this task from Backlog to Done on the Dumps-Generation board.Jul 20 2019, 5:23 AM