Page MenuHomePhabricator
Create Task
Maniphest T282145

ExternalStore makes primary read on GET requests
Closed, ResolvedPublicPRODUCTION ERROR
Actions

Description

ExternalStoreDB::batchFetchBlobs, used in all ExternalStore requests, falls back to the primary connection when the blobs are not found for some of the provided IDs. Callers have no control over this whatsoever. On GET requests it triggers warnings like the one above.

Either callers should be able to control this behavior or it should suppress warnings.

Example
normalized_message
Expectation (masterConns <=) 0 by MediaWiki::main not met (actual: {actual}):
{query}
exception.trace
from /srv/mediawiki/php-1.37.0-wmf.3/includes/libs/rdbms/TransactionProfiler.php(432)
#0 /srv/mediawiki/php-1.37.0-wmf.3/includes/libs/rdbms/TransactionProfiler.php(212): Wikimedia\Rdbms\TransactionProfiler->reportExpectationViolated(string, string, integer)
#1 /srv/mediawiki/php-1.37.0-wmf.3/includes/libs/rdbms/loadbalancer/LoadBalancer.php(1008): Wikimedia\Rdbms\TransactionProfiler->recordConnection(string, string, boolean)
#2 /srv/mediawiki/php-1.37.0-wmf.3/includes/libs/rdbms/loadbalancer/LoadBalancer.php(962): Wikimedia\Rdbms\LoadBalancer->getServerConnection(integer, string, integer)
#3 /srv/mediawiki/php-1.37.0-wmf.3/includes/libs/rdbms/loadbalancer/LoadBalancer.php(1128): Wikimedia\Rdbms\LoadBalancer->getConnection(integer, array, string, integer)
#4 /srv/mediawiki/php-1.37.0-wmf.3/includes/externalstore/ExternalStoreDB.php(197): Wikimedia\Rdbms\LoadBalancer->getMaintenanceConnectionRef(integer, array, string, integer)
#5 /srv/mediawiki/php-1.37.0-wmf.3/includes/externalstore/ExternalStoreDB.php(370): ExternalStoreDB->getMaster(string)
#6 /srv/mediawiki/php-1.37.0-wmf.3/includes/externalstore/ExternalStoreDB.php(95): ExternalStoreDB->batchFetchBlobs(string, array)
#7 /srv/mediawiki/php-1.37.0-wmf.3/includes/externalstore/ExternalStoreAccess.php(70): ExternalStoreDB->batchFetchFromURLs(array)
#8 /srv/mediawiki/php-1.37.0-wmf.3/includes/externalstore/ExternalStore.php(120): ExternalStoreAccess->fetchFromURLs(array)
#9 /srv/mediawiki/php-1.37.0-wmf.3/extensions/Flow/includes/Data/Utils/Merger.php(79): ExternalStore::batchFetchFromURLs(array)
#10 /srv/mediawiki/php-1.37.0-wmf.3/extensions/Flow/includes/Data/Storage/RevisionStorage.php(350): Flow\Data\Utils\Merger::mergeMulti(array, string, array, string, string)
#11 /srv/mediawiki/php-1.37.0-wmf.3/extensions/Flow/includes/Data/Storage/RevisionStorage.php(174): Flow\Data\Storage\RevisionStorage::mergeExternalContent(array)
#12 /srv/mediawiki/php-1.37.0-wmf.3/extensions/Flow/includes/Data/Index/FeatureIndex.php(468): Flow\Data\Storage\RevisionStorage->findMulti(array, array)
#13 /srv/mediawiki/php-1.37.0-wmf.3/extensions/Flow/includes/Data/Index/FeatureIndex.php(282): Flow\Data\Index\FeatureIndex->backingStoreFindMulti(array)
#14 /srv/mediawiki/php-1.37.0-wmf.3/extensions/Flow/includes/Data/ObjectLocator.php(95): Flow\Data\Index\FeatureIndex->findMulti(array, array)
#15 /srv/mediawiki/php-1.37.0-wmf.3/extensions/Flow/includes/Data/ObjectLocator.php(212): Flow\Data\ObjectLocator->findMulti(array)
#16 /srv/mediawiki/php-1.37.0-wmf.3/extensions/Flow/includes/Data/ManagerGroup.php(127): Flow\Data\ObjectLocator->getMulti(array)
#17 /srv/mediawiki/php-1.37.0-wmf.3/extensions/Flow/includes/Data/ManagerGroup.php(135): Flow\Data\ManagerGroup->call(string, array)
#18 /srv/mediawiki/php-1.37.0-wmf.3/extensions/Flow/includes/Formatter/ChangesListQuery.php(86): Flow\Data\ManagerGroup->getMulti(string, array)
#19 /srv/mediawiki/php-1.37.0-wmf.3/extensions/Flow/includes/Hooks.php(384): Flow\Formatter\ChangesListQuery->loadMetadataBatch(Wikimedia\Rdbms\ResultWrapper, boolean)
#20 /srv/mediawiki/php-1.37.0-wmf.3/includes/HookContainer/HookContainer.php(330): Flow\Hooks::onChangesListInitRows(OldChangesList, Wikimedia\Rdbms\ResultWrapper)
#21 /srv/mediawiki/php-1.37.0-wmf.3/includes/HookContainer/HookContainer.php(137): MediaWiki\HookContainer\HookContainer->callLegacyHook(string, array, array, array)
#22 /srv/mediawiki/php-1.37.0-wmf.3/includes/HookContainer/HookRunner.php(1173): MediaWiki\HookContainer\HookContainer->run(string, array)
#23 /srv/mediawiki/php-1.37.0-wmf.3/includes/changes/ChangesList.php(323): MediaWiki\HookContainer\HookRunner->onChangesListInitRows(OldChangesList, Wikimedia\Rdbms\ResultWrapper)
#24 /srv/mediawiki/php-1.37.0-wmf.3/includes/specials/SpecialRecentChanges.php(461): ChangesList->initChangesListRows(Wikimedia\Rdbms\ResultWrapper)
#25 /srv/mediawiki/php-1.37.0-wmf.3/includes/specialpage/ChangesListSpecialPage.php(1682): SpecialRecentChanges->outputChangesList(Wikimedia\Rdbms\ResultWrapper, FormOptions)
#26 /srv/mediawiki/php-1.37.0-wmf.3/includes/specialpage/ChangesListSpecialPage.php(681): ChangesListSpecialPage->webOutput(Wikimedia\Rdbms\ResultWrapper, FormOptions)
#27 /srv/mediawiki/php-1.37.0-wmf.3/includes/specials/SpecialRecentChanges.php(203): ChangesListSpecialPage->execute(NULL)
#28 /srv/mediawiki/php-1.37.0-wmf.3/includes/specialpage/SpecialPage.php(646): SpecialRecentChanges->execute(NULL)
#29 /srv/mediawiki/php-1.37.0-wmf.3/includes/specialpage/SpecialPageFactory.php(1397): SpecialPage->run(NULL)
#30 /srv/mediawiki/php-1.37.0-wmf.3/includes/MediaWiki.php(313): MediaWiki\SpecialPage\SpecialPageFactory->executePath(string, RequestContext)
#31 /srv/mediawiki/php-1.37.0-wmf.3/includes/MediaWiki.php(916): MediaWiki->performRequest()
#32 /srv/mediawiki/php-1.37.0-wmf.3/includes/MediaWiki.php(550): MediaWiki->main()
#33 /srv/mediawiki/php-1.37.0-wmf.3/index.php(53): MediaWiki->run()
#34 /srv/mediawiki/php-1.37.0-wmf.3/index.php(46): wfIndexMain()
#35 /srv/mediawiki/w/index.php(3): require(string)
#36 {main}

Details

SubjectRepoBranchLines +/-
Silence transaction profiler warnings in ExternalStoreDB::fetchBlob()mediawiki/coremaster+7 -2
Customize query in gerrit

Related Objects

Event Timeline

Tgr created this task.May 6 2021, 2:15 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 6 2021, 2:15 PM
Mainframe98 subscribed.May 6 2021, 3:22 PM
Krinkle added projects: Platform Engineering, Sustainability.May 12 2021, 6:43 PM
Krinkle moved this task from Untriaged to May 2021 on the Wikimedia-production-error board.
Krinkle moved this task from Untriaged to ExternalStore on the MediaWiki-Core-Revision-backend board.
tstarling subscribed.May 13 2021, 9:32 PM

Just need to call TransactionProfiler->setSilenced( true ) in ExternalStore before doing the query. There's a lot of random things doing that already. My understanding of these warnings is that they're supposed to uncover unknown, potentially high-traffic cases. We knew from the outset that there would be some things that need to connect to the master on a GET request.

gerritbot added a comment.May 16 2021, 4:17 PM

Change 691980 had a related patch set uploaded (by Gergő Tisza; author: Gergő Tisza):

[mediawiki/core@master] Silence transaction profiler warnings in ExternalStoreDB::fetchBlob()

https://gerrit.wikimedia.org/r/691980

gerritbot added a project: Patch-For-Review.May 16 2021, 4:17 PM
gerritbot added a comment.May 16 2021, 11:58 PM

Change 691980 merged by jenkins-bot:

[mediawiki/core@master] Silence transaction profiler warnings in ExternalStoreDB::fetchBlob()

https://gerrit.wikimedia.org/r/691980

Maintenance_bot removed a project: Patch-For-Review.May 17 2021, 12:10 AM
Tgr closed this task as Resolved.May 17 2021, 2:31 PM
Tgr claimed this task.
Krinkle mentioned this in Blog Post: Production Excellence #32: May 2021.Jun 21 2021, 1:31 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits