Page MenuHomePhabricator

10X increase in DBPerformance warnings on 1.34-wmf.10
Closed, ResolvedPublic

Description

Error

Request ID: XRUs2QpAEDQAADCE7BcAAABD

message
Expectation (masterConns <= 0) by ApiMain::setRequestExpectations not met (actual: 1):
[connect to 10.64.0.7 (enwiki)]
trace
#0 /srv/mediawiki/php-1.34.0-wmf.11/includes/libs/rdbms/TransactionProfiler.php(186): Wikimedia\Rdbms\TransactionProfiler->reportExpectationViolated()
#1 /srv/mediawiki/php-1.34.0-wmf.11/includes/libs/rdbms/loadbalancer/LoadBalancer.php(858): Wikimedia\Rdbms\TransactionProfiler->recordConnection()
#2 /srv/mediawiki/php-1.34.0-wmf.11/includes/libs/rdbms/loadbalancer/LoadBalancer.php(595): Wikimedia\Rdbms\LoadBalancer->getConnection()
#3 /srv/mediawiki/php-1.34.0-wmf.11/includes/libs/rdbms/loadbalancer/LoadBalancer.php(479): Wikimedia\Rdbms\LoadBalancer->pickReaderIndex()
#4 /srv/mediawiki/php-1.34.0-wmf.11/includes/libs/rdbms/loadbalancer/LoadBalancer.php(420): Wikimedia\Rdbms\LoadBalancer->getReaderIndex()
#5 /srv/mediawiki/php-1.34.0-wmf.11/includes/libs/rdbms/loadbalancer/LoadBalancer.php(840): Wikimedia\Rdbms\LoadBalancer->getConnectionIndex()
#6 /srv/mediawiki/php-1.34.0-wmf.11/includes/libs/rdbms/loadbalancer/LoadBalancer.php(944): Wikimedia\Rdbms\LoadBalancer->getConnection()
#7 /srv/mediawiki/php-1.34.0-wmf.11/includes/externalstore/ExternalStoreDB.php(150): Wikimedia\Rdbms\LoadBalancer->getConnectionRef()
#8 /srv/mediawiki/php-1.34.0-wmf.11/includes/externalstore/ExternalStoreDB.php(247): ExternalStoreDB->getSlave()
#9 /srv/mediawiki/php-1.34.0-wmf.11/includes/externalstore/ExternalStoreDB.php(49): ExternalStoreDB->fetchBlob()
#10 /srv/mediawiki/php-1.34.0-wmf.11/includes/externalstore/ExternalStore.php(86): ExternalStoreDB->fetchFromURL()
#11 /srv/mediawiki/php-1.34.0-wmf.11/includes/Storage/SqlBlobStore.php(415): ExternalStore::fetchFromURL()
#12 /srv/mediawiki/php-1.34.0-wmf.11/includes/libs/objectcache/WANObjectCache.php(1414): Closure$MediaWiki\Storage\SqlBlobStore::expandBlob()
#13 /srv/mediawiki/php-1.34.0-wmf.11/includes/libs/objectcache/WANObjectCache.php(1275): WANObjectCache->doGetWithSetCallback()
#14 /srv/mediawiki/php-1.34.0-wmf.11/includes/Storage/SqlBlobStore.php(420): WANObjectCache->getWithSetCallback()
#15 /srv/mediawiki/php-1.34.0-wmf.11/includes/Storage/SqlBlobStore.php(347): MediaWiki\Storage\SqlBlobStore->expandBlob()
#16 /srv/mediawiki/php-1.34.0-wmf.11/includes/Storage/SqlBlobStore.php(275): MediaWiki\Storage\SqlBlobStore->fetchBlob()
#17 /srv/mediawiki/php-1.34.0-wmf.11/includes/libs/objectcache/WANObjectCache.php(1414): Closure$MediaWiki\Storage\SqlBlobStore::getBlob()
#18 /srv/mediawiki/php-1.34.0-wmf.11/includes/libs/objectcache/WANObjectCache.php(1275): WANObjectCache->doGetWithSetCallback()
#19 /srv/mediawiki/php-1.34.0-wmf.11/includes/Storage/SqlBlobStore.php(278): WANObjectCache->getWithSetCallback()
#20 /srv/mediawiki/php-1.34.0-wmf.11/includes/Revision/RevisionStore.php(1462): MediaWiki\Storage\SqlBlobStore->getBlob()
#21 /srv/mediawiki/php-1.34.0-wmf.11/includes/Revision/RevisionStore.php(1637): MediaWiki\Revision\RevisionStore->loadSlotContent()
#22 (): Closure$MediaWiki\Revision\RevisionStore::loadSlotRecords()
#23 /srv/mediawiki/php-1.34.0-wmf.11/includes/Revision/SlotRecord.php(307): call_user_func()
#24 /srv/mediawiki/php-1.34.0-wmf.11/includes/Revision/RevisionRecord.php(176): MediaWiki\Revision\SlotRecord->getContent()
#25 /srv/mediawiki/php-1.34.0-wmf.11/includes/Revision.php(923): MediaWiki\Revision\RevisionRecord->getContent()
#26 /srv/mediawiki/php-1.34.0-wmf.11/includes/page/WikiPage.php(819): Revision->getContent()
#27 /srv/mediawiki/php-1.34.0-wmf.11/includes/api/ApiParse.php(574): WikiPage->getContent()
#28 /srv/mediawiki/php-1.34.0-wmf.11/includes/api/ApiParse.php(164): ApiParse->getParsedContent()
#29 /srv/mediawiki/php-1.34.0-wmf.11/includes/api/ApiMain.php(1583): ApiParse->execute()
#30 /srv/mediawiki/php-1.34.0-wmf.11/includes/api/ApiMain.php(500): ApiMain->executeAction()
#31 /srv/mediawiki/php-1.34.0-wmf.11/extensions/TextExtracts/includes/ApiQueryExtracts.php(235): ApiMain->execute()
#32 /srv/mediawiki/php-1.34.0-wmf.11/extensions/TextExtracts/includes/ApiQueryExtracts.php(159): TextExtracts\ApiQueryExtracts->parse()
#33 /srv/mediawiki/php-1.34.0-wmf.11/extensions/TextExtracts/includes/ApiQueryExtracts.php(96): TextExtracts\ApiQueryExtracts->getExtract()
#34 /srv/mediawiki/php-1.34.0-wmf.11/includes/api/ApiQuery.php(250): TextExtracts\ApiQueryExtracts->execute()
#35 /srv/mediawiki/php-1.34.0-wmf.11/includes/api/ApiMain.php(1583): ApiQuery->execute()
#36 /srv/mediawiki/php-1.34.0-wmf.11/includes/api/ApiMain.php(500): ApiMain->executeAction()
#37 /srv/mediawiki/php-1.34.0-wmf.11/extensions/TextExtracts/includes/Hooks.php(37): ApiMain->execute()
#38 /srv/mediawiki/php-1.34.0-wmf.11/includes/Hooks.php(174): TextExtracts\Hooks::onApiOpenSearchSuggest()
#39 /srv/mediawiki/php-1.34.0-wmf.11/includes/Hooks.php(202): Hooks::callHook()
#40 /srv/mediawiki/php-1.34.0-wmf.11/includes/api/ApiOpenSearch.php(94): Hooks::run()
#41 /srv/mediawiki/php-1.34.0-wmf.11/includes/api/ApiMain.php(1583): ApiOpenSearch->execute()
#42 /srv/mediawiki/php-1.34.0-wmf.11/includes/api/ApiMain.php(531): ApiMain->executeAction()
#43 /srv/mediawiki/php-1.34.0-wmf.11/includes/api/ApiMain.php(502): ApiMain->executeActionWithErrorHandling()
#44 /srv/mediawiki/php-1.34.0-wmf.11/api.php(87): ApiMain->execute()
#45 /srv/mediawiki/w/api.php(3): include()

Impact

(To be determined.)

Notes

DBPerformance (overall)
DBPerformance messages like this
DBPerformance messages not like this

Event Timeline

Krinkle created this task.Jun 27 2019, 8:52 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 27 2019, 8:52 PM
Krinkle updated the task description. (Show Details)Jun 27 2019, 8:54 PM

Change 519565 had a related patch set uploaded (by Aaron Schulz; owner: Aaron Schulz):
[mediawiki/core@master] rdbms: avoid recursion in LoadBalancer when the master has non-zero load

https://gerrit.wikimedia.org/r/519565

aaron claimed this task.Jul 1 2019, 7:50 PM
kchapman moved this task from Inbox to Doing on the Performance-Team board.Jul 1 2019, 7:51 PM

Change 521573 had a related patch set uploaded (by Krinkle; owner: Aaron Schulz):
[mediawiki/core@wmf/1.34.0-wmf.13] rdbms: avoid recursion in LoadBalancer when the master has non-zero load

https://gerrit.wikimedia.org/r/521573

Change 519565 merged by jenkins-bot:
[mediawiki/core@master] rdbms: avoid recursion in LoadBalancer when the master has non-zero load

https://gerrit.wikimedia.org/r/519565

Change 521573 merged by jenkins-bot:
[mediawiki/core@wmf/1.34.0-wmf.13] rdbms: avoid recursion in LoadBalancer when the master has non-zero load

https://gerrit.wikimedia.org/r/521573

Krinkle closed this task as Resolved.Jul 9 2019, 9:53 PM
Krinkle triaged this task as High priority.
Krinkle removed a project: Patch-For-Review.

Mentioned in SAL (#wikimedia-operations) [2019-07-09T21:53:26Z] <krinkle@deploy1001> Synchronized php-1.34.0-wmf.13/includes/libs/rdbms/: T226770 / 4c2a58589f2db (duration: 00m 59s)