Page MenuHomePhabricator
Create Task
Maniphest T228104

Wikibase dump scripts fail on external storage access
Closed, ResolvedPublicPRODUCTION ERROR
Actions

Description

Wikimedia\Rdbms\DBConnectionError from line 1392 of /srv/mediawiki/php-1.34.0-wmf.13/includes/libs/rdbms/loadbalancer/LoadBalancer.php: Cannot access the database: No working replica DB server: Unknown error
#0 /srv/mediawiki/php-1.34.0-wmf.13/includes/libs/rdbms/loadbalancer/LoadBalancer.php(467): Wikimedia\Rdbms\LoadBalancer->reportConnectionError()
#1 /srv/mediawiki/php-1.34.0-wmf.13/includes/libs/rdbms/loadbalancer/LoadBalancer.php(897): Wikimedia\Rdbms\LoadBalancer->getConnectionIndex(false, Array, 'wikidatawiki')
#2 /srv/mediawiki/php-1.34.0-wmf.13/includes/libs/rdbms/loadbalancer/LoadBalancer.php(1035): Wikimedia\Rdbms\LoadBalancer->getConnection(-1, Array, 'wikidatawiki', 0)
#3 /srv/mediawiki/php-1.34.0-wmf.13/includes/externalstore/ExternalStoreDB.php(166): Wikimedia\Rdbms\LoadBalancer->getConnectionRef(-1, Array, 'wikidatawiki')
#4 /srv/mediawiki/php-1.34.0-wmf.13/includes/externalstore/ExternalStoreDB.php(259): ExternalStoreDB->getSlave('cluster25')
#5 /srv/mediawiki/php-1.34.0-wmf.13/includes/externalstore/ExternalStoreDB.php(65): ExternalStoreDB->fetchBlob('cluster25', '487121788', false)
#6 /srv/mediawiki/php-1.34.0-wmf.13/includes/externalstore/ExternalStoreAccess.php(52): ExternalStoreDB->fetchFromURL('DB://cluster25/...')
#7 /srv/mediawiki/php-1.34.0-wmf.13/includes/Storage/SqlBlobStore.php(427): ExternalStoreAccess->fetchFromURL('DB://cluster25/...', Array)
#8 /srv/mediawiki/php-1.34.0-wmf.13/includes/libs/objectcache/WANObjectCache.php(1406): MediaWiki\Storage\SqlBlobStore->MediaWiki\Storage\{closure}(false, 604800, Array, NULL)
#9 /srv/mediawiki/php-1.34.0-wmf.13/includes/libs/objectcache/WANObjectCache.php(1269): WANObjectCache->doGetWithSetCallback('global:BlobStor...', 604800, Object(Closure), Array)
#10 /srv/mediawiki/php-1.34.0-wmf.13/includes/Storage/SqlBlobStore.php(431): WANObjectCache->getWithSetCallback('global:BlobStor...', 604800, Object(Closure), Array)
#11 /srv/mediawiki/php-1.34.0-wmf.13/includes/Storage/SqlBlobStore.php(358): MediaWiki\Storage\SqlBlobStore->expandBlob('DB://cluster25/...', Array, 'tt:971848740')
#12 /srv/mediawiki/php-1.34.0-wmf.13/includes/Storage/SqlBlobStore.php(286): MediaWiki\Storage\SqlBlobStore->fetchBlob('tt:971848740', 0)
#13 /srv/mediawiki/php-1.34.0-wmf.13/includes/libs/objectcache/WANObjectCache.php(1406): MediaWiki\Storage\SqlBlobStore->MediaWiki\Storage\{closure}(false, 604800, Array, NULL)
#14 /srv/mediawiki/php-1.34.0-wmf.13/includes/libs/objectcache/WANObjectCache.php(1269): WANObjectCache->doGetWithSetCallback('global:BlobStor...', 604800, Object(Closure), Array)
#15 /srv/mediawiki/php-1.34.0-wmf.13/includes/Storage/SqlBlobStore.php(288): WANObjectCache->getWithSetCallback('global:BlobStor...', 604800, Object(Closure), Array)
#16 /srv/mediawiki/php-1.34.0-wmf.13/extensions/Wikibase/lib/includes/Store/Sql/WikiPageEntityRevisionLookup.php(241): MediaWiki\Storage\SqlBlobStore->getBlob('tt:971848740', 0)
#17 /srv/mediawiki/php-1.34.0-wmf.13/extensions/Wikibase/lib/includes/Store/Sql/WikiPageEntityRevisionLookup.php(132): Wikibase\Lib\Store\Sql\WikiPageEntityRevisionLookup->loadEntity(Object(stdClass), 'replica')
#18 /srv/mediawiki/php-1.34.0-wmf.13/extensions/Wikibase/lib/includes/Store/DispatchingEntityRevisionLookup.php(59): Wikibase\Lib\Store\Sql\WikiPageEntityRevisionLookup->getEntityRevision(Object(Wikibase\DataModel\Entity\ItemId), 0, 'replica')
#19 /srv/mediawiki/php-1.34.0-wmf.13/extensions/Wikibase/lib/includes/Store/TypeDispatchingEntityRevisionLookup.php(54): Wikibase\Lib\Store\DispatchingEntityRevisionLookup->getEntityRevision(Object(Wikibase\DataModel\Entity\ItemId), 0, 'replica')
#20 /srv/mediawiki/php-1.34.0-wmf.13/extensions/Wikibase/lib/includes/Store/CacheRetrievingEntityRevisionLookup.php(75): Wikibase\Lib\Store\TypeDispatchingEntityRevisionLookup->getEntityRevision(Object(Wikibase\DataModel\Entity\ItemId), 0, 'replica')
#21 /srv/mediawiki/php-1.34.0-wmf.13/extensions/Wikibase/repo/includes/Dumpers/JsonDumpGenerator.php(128): Wikibase\Lib\Store\CacheRetrievingEntityRevisionLookup->getEntityRevision(Object(Wikibase\DataModel\Entity\ItemId))
#22 /srv/mediawiki/php-1.34.0-wmf.13/extensions/Wikibase/repo/includes/Dumpers/DumpGenerator.php(299): Wikibase\Dumpers\JsonDumpGenerator->generateDumpForEntityId(Object(Wikibase\DataModel\Entity\ItemId))
#23 /srv/mediawiki/php-1.34.0-wmf.13/extensions/Wikibase/repo/includes/Dumpers/DumpGenerator.php(262): Wikibase\Dumpers\DumpGenerator->dumpEntities(Array, 879)
#24 /srv/mediawiki/php-1.34.0-wmf.13/extensions/Wikibase/repo/maintenance/DumpEntities.php(221): Wikibase\Dumpers\DumpGenerator->generateDump(Object(Wikibase\Repo\Store\Sql\SqlEntityIdPager))
#25 /srv/mediawiki/php-1.34.0-wmf.13/extensions/Wikibase/repo/maintenance/dumpJson.php(100): Wikibase\DumpEntities->execute()
#26 /srv/mediawiki/php-1.34.0-wmf.13/maintenance/doMaintenance.php(99): Wikibase\DumpJson->execute()
#27 /srv/mediawiki/php-1.34.0-wmf.13/extensions/Wikibase/repo/maintenance/dumpJson.php(126): require_once('/srv/mediawiki/...')
#28 /srv/mediawiki/multiversion/MWScript.php(101): require_once('/srv/mediawiki/...')
#29 {main}

Details

SubjectRepoBranchLines +/-
rdbms: better handle a non-existing "defaultGroup" in LoadBalancermediawiki/corewmf/1.34.0-wmf.13+87 -82
rdbms: better handle a non-existing "defaultGroup" in LoadBalancermediawiki/corewmf/1.34.0-wmf.14+83 -82
LoadBalancer::getConnectionIndex: Also fallback to generic groupmediawiki/coremaster+33 -3
Customize query in gerrit

Related Objects

Event Timeline

hoo created this task.Jul 15 2019, 9:01 PM
Restricted Application added a project: Wikidata. · View Herald TranscriptJul 15 2019, 9:01 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
hoo added a comment.Jul 15 2019, 9:06 PM

This is again related to dbgroupdefault (caused by ab09cf2fa59).

hoo@mwmaint1002:~$ mwscript eval.php --wiki wikidatawiki
> $mediaWikiServices = MediaWiki\MediaWikiServices::getInstance(); $exStore = $mediaWikiServices->getExternalStoreAccess(); $exStore->fetchFromURL('DB://cluster25/487121788');

> ^D
hoo@mwmaint1002:~$ mwscript eval.php --wiki wikidatawiki --dbgroupdefault dumps
> $mediaWikiServices = MediaWiki\MediaWikiServices::getInstance(); $exStore = $mediaWikiServices->getExternalStoreAccess(); $exStore->fetchFromURL('DB://cluster25/487121788');
Caught exception Wikimedia\Rdbms\DBConnectionError: Cannot access the database: No working replica DB server: Unknown error
#0 /srv/mediawiki/php-1.34.0-wmf.13/includes/libs/rdbms/loadbalancer/LoadBalancer.php(467): Wikimedia\Rdbms\LoadBalancer->reportConnectionError()
#1 /srv/mediawiki/php-1.34.0-wmf.13/includes/libs/rdbms/loadbalancer/LoadBalancer.php(897): Wikimedia\Rdbms\LoadBalancer->getConnectionIndex()
#2 /srv/mediawiki/php-1.34.0-wmf.13/includes/libs/rdbms/loadbalancer/LoadBalancer.php(1035): Wikimedia\Rdbms\LoadBalancer->getConnection()
#3 /srv/mediawiki/php-1.34.0-wmf.13/includes/externalstore/ExternalStoreDB.php(166): Wikimedia\Rdbms\LoadBalancer->getConnectionRef()
#4 /srv/mediawiki/php-1.34.0-wmf.13/includes/externalstore/ExternalStoreDB.php(259): ExternalStoreDB->getSlave()
#5 /srv/mediawiki/php-1.34.0-wmf.13/includes/externalstore/ExternalStoreDB.php(65): ExternalStoreDB->fetchBlob()
#6 /srv/mediawiki/php-1.34.0-wmf.13/includes/externalstore/ExternalStoreAccess.php(52): ExternalStoreDB->fetchFromURL()
#7 /srv/mediawiki/php-1.34.0-wmf.13/maintenance/eval.php(78)(57f4d4e974a5dc7047f3cca4b7fdb441) : eval()'d code(1): ExternalStoreAccess->fetchFromURL()
#8 /srv/mediawiki/php-1.34.0-wmf.13/maintenance/eval.php(78): include()
#9 /srv/mediawiki/multiversion/MWScript.php(101): include()
#10 {main}
>
gerritbot added a comment.Jul 15 2019, 9:12 PM

Change 521920 had a related patch set uploaded (by Hoo man; owner: Hoo man):
[mediawiki/core@master] LoadBalancer::getConnectionIndex: Also fallback to generic group

https://gerrit.wikimedia.org/r/521920

gerritbot added a project: Patch-For-Review.Jul 15 2019, 9:12 PM
Lydia_Pintscher added a project: Wikidata-Campsite.Jul 16 2019, 9:37 AM
Lydia_Pintscher moved this task from Incoming to Ready to estimate on the Wikidata-Campsite board.
ArielGlenn added a project: Dumps-Generation.Jul 16 2019, 9:42 AM
ArielGlenn subscribed.
hoo added a comment.Jul 16 2019, 11:30 AM

Alternative patch: https://gerrit.wikimedia.org/r/c/mediawiki/core/+/523412

• alaa_wmde edited projects, added Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)); removed Wikidata-Campsite.Jul 16 2019, 12:33 PM
• alaa_wmde moved this task from To Do (prioritised from top to bottom) to Peer Review on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.
Krinkle moved this task from Untriaged to Jul2019/1.34.wmf.13+ on the Wikimedia-production-error board.Jul 16 2019, 2:38 PM
gerritbot added a comment.Jul 17 2019, 7:08 PM

Change 521920 abandoned by Hoo man:
LoadBalancer::getConnectionIndex: Also fallback to generic group

Reason:
Better solved through https://gerrit.wikimedia.org/r/c/mediawiki/core/ /523412

https://gerrit.wikimedia.org/r/521920

Maintenance_bot removed a project: Patch-For-Review.Jul 17 2019, 7:10 PM
hoo moved this task from Peer Review to Test (Verification) on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.Jul 17 2019, 7:10 PM
gerritbot added a comment.Jul 17 2019, 7:15 PM

Change 523990 had a related patch set uploaded (by Jforrester; owner: Aaron Schulz):
[mediawiki/core@wmf/1.34.0-wmf.14] rdbms: better handle a non-existing "defaultGroup" in LoadBalancer

https://gerrit.wikimedia.org/r/523990

gerritbot added a project: Patch-For-Review.Jul 17 2019, 7:15 PM
gerritbot added a comment.Jul 17 2019, 7:27 PM

Change 523990 merged by jenkins-bot:
[mediawiki/core@wmf/1.34.0-wmf.14] rdbms: better handle a non-existing "defaultGroup" in LoadBalancer

https://gerrit.wikimedia.org/r/523990

Stashbot added a comment.Jul 17 2019, 7:35 PM

Mentioned in SAL (#wikimedia-operations) [2019-07-17T19:35:56Z] <jforrester@deploy1001> Synchronized php-1.34.0-wmf.14/includes/libs/rdbms/loadbalancer: T228104 rdbms: better handle a non-existing defaultGroup in LoadBalancer (duration: 00m 55s)

gerritbot added a comment.Jul 17 2019, 7:41 PM

Change 523997 had a related patch set uploaded (by Jforrester; owner: Aaron Schulz):
[mediawiki/core@wmf/1.34.0-wmf.13] rdbms: better handle a non-existing "defaultGroup" in LoadBalancer

https://gerrit.wikimedia.org/r/523997

hoo added a comment.Jul 17 2019, 7:42 PM

We backported this to wmf14 now to check for fallout. If there's none, we will also target wmf13 later on.

ReleaseTaggerBot added a project: MW-1.34-notes (1.34.0-wmf.14; 2019-07-16).Jul 17 2019, 8:00 PM
gerritbot added a comment.Jul 17 2019, 9:12 PM

Change 523997 merged by jenkins-bot:
[mediawiki/core@wmf/1.34.0-wmf.13] rdbms: better handle a non-existing "defaultGroup" in LoadBalancer

https://gerrit.wikimedia.org/r/523997

Stashbot added a comment.Jul 17 2019, 9:16 PM

Mentioned in SAL (#wikimedia-operations) [2019-07-17T21:16:52Z] <jforrester@deploy1001> Synchronized php-1.34.0-wmf.13/includes/libs/rdbms/loadbalancer: T228104 rdbms: better handle a non-existing defaultGroup in LoadBalancer (duration: 00m 55s)

Jdforrester-WMF subscribed.Jul 17 2019, 9:29 PM

This is now deployed to wmf.13. Can we trigger a dumps run to show that it works?

ArielGlenn added a comment.Jul 17 2019, 9:45 PM

I have started the json format dumps and they seem to be running correctly.

Envlh subscribed.Jul 17 2019, 9:57 PM
ReleaseTaggerBot edited projects, added MW-1.34-notes (1.34.0-wmf.13; 2019-07-09); removed MW-1.34-notes (1.34.0-wmf.14; 2019-07-16).Jul 17 2019, 10:00 PM
Maintenance_bot removed a project: Patch-For-Review.Jul 17 2019, 10:10 PM
Stashbot added a comment.Jul 17 2019, 10:16 PM

Mentioned in SAL (#wikimedia-operations) [2019-07-17T22:16:22Z] <hoo> Manually started the Wikidata RDF dumps on snapshot1008 (due to T228104)

ArielGlenn closed this task as Resolved.Jul 19 2019, 7:36 AM
ArielGlenn claimed this task.

The above changes were deployed everywhere on Wednesday. JSON and Rdf dumps have been running since then without issues. The JSON dumps should complete sometime today. Closing this, whew!

Envlh added a comment.Jul 19 2019, 4:56 PM

Thank you all! :)

ArielGlenn moved this task from Backlog to Done on the Dumps-Generation board.Jul 20 2019, 5:23 AM
ArielGlenn mentioned this in T228569: Suspicious data loss on the Query Service.Jul 21 2019, 8:54 PM
• alaa_wmde moved this task from Test (Verification) to Done on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.Jul 26 2019, 1:14 PM
• mmodell changed the subtype of this task from "Task" to "Production Error".Aug 28 2019, 11:05 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits