Page MenuHomePhabricator
Create Task
Maniphest T226770

10X increase in DBPerformance warnings on 1.34-wmf.10
Closed, ResolvedPublicPRODUCTION ERROR
Actions

Assigned To
aaron
Authored By
Krinkle
Jun 27 2019, 8:52 PM
Tags
Referenced Files
F29658257: Screenshot 2019-06-27 at 21.51.38.png
Jun 27 2019, 8:52 PM
F29658261: Screenshot 2019-06-27 at 21.51.10.png
Jun 27 2019, 8:52 PM
F29658253: Screenshot 2019-06-27 at 21.51.44.png
Jun 27 2019, 8:52 PM
Subscribers
Aklapper
Krinkle

Description

Error

Request ID: XRUs2QpAEDQAADCE7BcAAABD

message
Expectation (masterConns <= 0) by ApiMain::setRequestExpectations not met (actual: 1):
[connect to 10.64.0.7 (enwiki)]
trace
#0 /srv/mediawiki/php-1.34.0-wmf.11/includes/libs/rdbms/TransactionProfiler.php(186): Wikimedia\Rdbms\TransactionProfiler->reportExpectationViolated()
#1 /srv/mediawiki/php-1.34.0-wmf.11/includes/libs/rdbms/loadbalancer/LoadBalancer.php(858): Wikimedia\Rdbms\TransactionProfiler->recordConnection()
#2 /srv/mediawiki/php-1.34.0-wmf.11/includes/libs/rdbms/loadbalancer/LoadBalancer.php(595): Wikimedia\Rdbms\LoadBalancer->getConnection()
#3 /srv/mediawiki/php-1.34.0-wmf.11/includes/libs/rdbms/loadbalancer/LoadBalancer.php(479): Wikimedia\Rdbms\LoadBalancer->pickReaderIndex()
#4 /srv/mediawiki/php-1.34.0-wmf.11/includes/libs/rdbms/loadbalancer/LoadBalancer.php(420): Wikimedia\Rdbms\LoadBalancer->getReaderIndex()
#5 /srv/mediawiki/php-1.34.0-wmf.11/includes/libs/rdbms/loadbalancer/LoadBalancer.php(840): Wikimedia\Rdbms\LoadBalancer->getConnectionIndex()
#6 /srv/mediawiki/php-1.34.0-wmf.11/includes/libs/rdbms/loadbalancer/LoadBalancer.php(944): Wikimedia\Rdbms\LoadBalancer->getConnection()
#7 /srv/mediawiki/php-1.34.0-wmf.11/includes/externalstore/ExternalStoreDB.php(150): Wikimedia\Rdbms\LoadBalancer->getConnectionRef()
#8 /srv/mediawiki/php-1.34.0-wmf.11/includes/externalstore/ExternalStoreDB.php(247): ExternalStoreDB->getSlave()
#9 /srv/mediawiki/php-1.34.0-wmf.11/includes/externalstore/ExternalStoreDB.php(49): ExternalStoreDB->fetchBlob()
#10 /srv/mediawiki/php-1.34.0-wmf.11/includes/externalstore/ExternalStore.php(86): ExternalStoreDB->fetchFromURL()
#11 /srv/mediawiki/php-1.34.0-wmf.11/includes/Storage/SqlBlobStore.php(415): ExternalStore::fetchFromURL()
#12 /srv/mediawiki/php-1.34.0-wmf.11/includes/libs/objectcache/WANObjectCache.php(1414): Closure$MediaWiki\Storage\SqlBlobStore::expandBlob()
#13 /srv/mediawiki/php-1.34.0-wmf.11/includes/libs/objectcache/WANObjectCache.php(1275): WANObjectCache->doGetWithSetCallback()
#14 /srv/mediawiki/php-1.34.0-wmf.11/includes/Storage/SqlBlobStore.php(420): WANObjectCache->getWithSetCallback()
#15 /srv/mediawiki/php-1.34.0-wmf.11/includes/Storage/SqlBlobStore.php(347): MediaWiki\Storage\SqlBlobStore->expandBlob()
#16 /srv/mediawiki/php-1.34.0-wmf.11/includes/Storage/SqlBlobStore.php(275): MediaWiki\Storage\SqlBlobStore->fetchBlob()
#17 /srv/mediawiki/php-1.34.0-wmf.11/includes/libs/objectcache/WANObjectCache.php(1414): Closure$MediaWiki\Storage\SqlBlobStore::getBlob()
#18 /srv/mediawiki/php-1.34.0-wmf.11/includes/libs/objectcache/WANObjectCache.php(1275): WANObjectCache->doGetWithSetCallback()
#19 /srv/mediawiki/php-1.34.0-wmf.11/includes/Storage/SqlBlobStore.php(278): WANObjectCache->getWithSetCallback()
#20 /srv/mediawiki/php-1.34.0-wmf.11/includes/Revision/RevisionStore.php(1462): MediaWiki\Storage\SqlBlobStore->getBlob()
#21 /srv/mediawiki/php-1.34.0-wmf.11/includes/Revision/RevisionStore.php(1637): MediaWiki\Revision\RevisionStore->loadSlotContent()
#22 (): Closure$MediaWiki\Revision\RevisionStore::loadSlotRecords()
#23 /srv/mediawiki/php-1.34.0-wmf.11/includes/Revision/SlotRecord.php(307): call_user_func()
#24 /srv/mediawiki/php-1.34.0-wmf.11/includes/Revision/RevisionRecord.php(176): MediaWiki\Revision\SlotRecord->getContent()
#25 /srv/mediawiki/php-1.34.0-wmf.11/includes/Revision.php(923): MediaWiki\Revision\RevisionRecord->getContent()
#26 /srv/mediawiki/php-1.34.0-wmf.11/includes/page/WikiPage.php(819): Revision->getContent()
#27 /srv/mediawiki/php-1.34.0-wmf.11/includes/api/ApiParse.php(574): WikiPage->getContent()
#28 /srv/mediawiki/php-1.34.0-wmf.11/includes/api/ApiParse.php(164): ApiParse->getParsedContent()
#29 /srv/mediawiki/php-1.34.0-wmf.11/includes/api/ApiMain.php(1583): ApiParse->execute()
#30 /srv/mediawiki/php-1.34.0-wmf.11/includes/api/ApiMain.php(500): ApiMain->executeAction()
#31 /srv/mediawiki/php-1.34.0-wmf.11/extensions/TextExtracts/includes/ApiQueryExtracts.php(235): ApiMain->execute()
#32 /srv/mediawiki/php-1.34.0-wmf.11/extensions/TextExtracts/includes/ApiQueryExtracts.php(159): TextExtracts\ApiQueryExtracts->parse()
#33 /srv/mediawiki/php-1.34.0-wmf.11/extensions/TextExtracts/includes/ApiQueryExtracts.php(96): TextExtracts\ApiQueryExtracts->getExtract()
#34 /srv/mediawiki/php-1.34.0-wmf.11/includes/api/ApiQuery.php(250): TextExtracts\ApiQueryExtracts->execute()
#35 /srv/mediawiki/php-1.34.0-wmf.11/includes/api/ApiMain.php(1583): ApiQuery->execute()
#36 /srv/mediawiki/php-1.34.0-wmf.11/includes/api/ApiMain.php(500): ApiMain->executeAction()
#37 /srv/mediawiki/php-1.34.0-wmf.11/extensions/TextExtracts/includes/Hooks.php(37): ApiMain->execute()
#38 /srv/mediawiki/php-1.34.0-wmf.11/includes/Hooks.php(174): TextExtracts\Hooks::onApiOpenSearchSuggest()
#39 /srv/mediawiki/php-1.34.0-wmf.11/includes/Hooks.php(202): Hooks::callHook()
#40 /srv/mediawiki/php-1.34.0-wmf.11/includes/api/ApiOpenSearch.php(94): Hooks::run()
#41 /srv/mediawiki/php-1.34.0-wmf.11/includes/api/ApiMain.php(1583): ApiOpenSearch->execute()
#42 /srv/mediawiki/php-1.34.0-wmf.11/includes/api/ApiMain.php(531): ApiMain->executeAction()
#43 /srv/mediawiki/php-1.34.0-wmf.11/includes/api/ApiMain.php(502): ApiMain->executeActionWithErrorHandling()
#44 /srv/mediawiki/php-1.34.0-wmf.11/api.php(87): ApiMain->execute()
#45 /srv/mediawiki/w/api.php(3): include()

Impact

(To be determined.)

Notes

DBPerformance (overall)
Screenshot 2019-06-27 at 21.51.10.png (434×1 px, 84 KB)
DBPerformance messages like this
Screenshot 2019-06-27 at 21.51.38.png (516×1 px, 92 KB)
DBPerformance messages not like this
Screenshot 2019-06-27 at 21.51.44.png (496×1 px, 103 KB)

Details

SubjectRepoBranchLines +/-
rdbms: avoid recursion in LoadBalancer when the master has non-zero loadmediawiki/corewmf/1.34.0-wmf.13+153 -79
rdbms: avoid recursion in LoadBalancer when the master has non-zero loadmediawiki/coremaster+153 -79
Customize query in gerrit

Related Objects

Event Timeline

Krinkle created this task.Jun 27 2019, 8:52 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 27 2019, 8:52 PM
Krinkle updated the task description. (Show Details)Jun 27 2019, 8:54 PM
Krinkle moved this task from Untriaged to Rdbms library on the MediaWiki-libs-Rdbms board.Jun 27 2019, 9:06 PM
gerritbot added a comment.Jun 28 2019, 1:39 AM

Change 519565 had a related patch set uploaded (by Aaron Schulz; owner: Aaron Schulz):
[mediawiki/core@master] rdbms: avoid recursion in LoadBalancer when the master has non-zero load

https://gerrit.wikimedia.org/r/519565

gerritbot added a project: Patch-For-Review.Jun 28 2019, 1:39 AM
aaron claimed this task.Jul 1 2019, 7:50 PM
• kchapman moved this task from Inbox, needs triage to Doing (old) on the Performance-Team board.Jul 1 2019, 7:51 PM
gerritbot added a comment.Jul 9 2019, 7:28 PM

Change 521573 had a related patch set uploaded (by Krinkle; owner: Aaron Schulz):
[mediawiki/core@wmf/1.34.0-wmf.13] rdbms: avoid recursion in LoadBalancer when the master has non-zero load

https://gerrit.wikimedia.org/r/521573

gerritbot added a comment.Jul 9 2019, 7:49 PM

Change 519565 merged by jenkins-bot:
[mediawiki/core@master] rdbms: avoid recursion in LoadBalancer when the master has non-zero load

https://gerrit.wikimedia.org/r/519565

ReleaseTaggerBot added a project: MW-1.34-notes (1.34.0-wmf.14; 2019-07-16).Jul 9 2019, 8:00 PM
Krinkle moved this task from Untriaged to Jun 2019 / 1.34.wmf.8+ on the Wikimedia-production-error board.Jul 9 2019, 9:31 PM
gerritbot added a comment.Jul 9 2019, 9:38 PM

Change 521573 merged by jenkins-bot:
[mediawiki/core@wmf/1.34.0-wmf.13] rdbms: avoid recursion in LoadBalancer when the master has non-zero load

https://gerrit.wikimedia.org/r/521573

Krinkle closed this task as Resolved.Jul 9 2019, 9:53 PM
Krinkle triaged this task as High priority.
Krinkle removed a project: Patch-For-Review.
Stashbot added a comment.Jul 9 2019, 9:53 PM

Mentioned in SAL (#wikimedia-operations) [2019-07-09T21:53:26Z] <krinkle@deploy1001> Synchronized php-1.34.0-wmf.13/includes/libs/rdbms/: T226770 / rMW4c2a58589f2d (duration: 00m 59s)

ReleaseTaggerBot edited projects, added MW-1.34-notes (1.34.0-wmf.13; 2019-07-09); removed MW-1.34-notes (1.34.0-wmf.14; 2019-07-16).Jul 9 2019, 10:00 PM
• mmodell changed the subtype of this task from "Task" to "Production Error".Aug 28 2019, 11:06 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL