Page MenuHomePhabricator
Create Task
Maniphest T282605

Special:Contributions with Flow entries throw "Timed out waiting for replication to reach" (via ExternalStoreDB)
Open, Needs TriagePublicPRODUCTION ERROR
Actions

Description

Error
normalized_message
Timed out waiting for replication to reach {wait_pos}
exception.trace
from /srv/mediawiki/php-1.37.0-wmf.4/includes/libs/rdbms/database/DatabaseMysqlBase.php(935)
#0 /srv/mediawiki/php-1.37.0-wmf.4/includes/libs/rdbms/loadbalancer/LoadBalancer.php(937): Wikimedia\Rdbms\DatabaseMysqlBase->masterPosWait(Wikimedia\Rdbms\MySQLMasterPos, integer)
#1 /srv/mediawiki/php-1.37.0-wmf.4/includes/libs/rdbms/loadbalancer/LoadBalancer.php(565): Wikimedia\Rdbms\LoadBalancer->doWait(integer)
#2 /srv/mediawiki/php-1.37.0-wmf.4/includes/libs/rdbms/loadbalancer/LoadBalancer.php(508): Wikimedia\Rdbms\LoadBalancer->getReaderIndex(string, string)
#3 /srv/mediawiki/php-1.37.0-wmf.4/includes/libs/rdbms/loadbalancer/LoadBalancer.php(960): Wikimedia\Rdbms\LoadBalancer->getConnectionIndex(integer, array, string)
#4 /srv/mediawiki/php-1.37.0-wmf.4/includes/libs/rdbms/loadbalancer/LoadBalancer.php(1101): Wikimedia\Rdbms\LoadBalancer->getConnection(integer, array, string, integer)
#5 /srv/mediawiki/php-1.37.0-wmf.4/includes/externalstore/ExternalStoreDB.php(168): Wikimedia\Rdbms\LoadBalancer->getConnectionRef(integer, array, string, integer)
#6 /srv/mediawiki/php-1.37.0-wmf.4/includes/externalstore/ExternalStoreDB.php(352): ExternalStoreDB->getReplica(string)
#7 /srv/mediawiki/php-1.37.0-wmf.4/includes/externalstore/ExternalStoreDB.php(95): ExternalStoreDB->batchFetchBlobs(string, array)
#8 /srv/mediawiki/php-1.37.0-wmf.4/includes/externalstore/ExternalStoreAccess.php(70): ExternalStoreDB->batchFetchFromURLs(array)
#9 /srv/mediawiki/php-1.37.0-wmf.4/includes/externalstore/ExternalStore.php(120): ExternalStoreAccess->fetchFromURLs(array)
#10 /srv/mediawiki/php-1.37.0-wmf.4/extensions/Flow/includes/Data/Utils/Merger.php(79): ExternalStore::batchFetchFromURLs(array)
#11 /srv/mediawiki/php-1.37.0-wmf.4/extensions/Flow/includes/Data/Storage/RevisionStorage.php(350): Flow\Data\Utils\Merger::mergeMulti(array, string, array, string, string)
#12 /srv/mediawiki/php-1.37.0-wmf.4/extensions/Flow/includes/Formatter/ContributionsQuery.php(272): Flow\Data\Storage\RevisionStorage::mergeExternalContent(array)
#13 /srv/mediawiki/php-1.37.0-wmf.4/extensions/Flow/includes/Formatter/ContributionsQuery.php(80): Flow\Formatter\ContributionsQuery->loadRevisions(Wikimedia\Rdbms\ResultWrapper, string)
#14 /srv/mediawiki/php-1.37.0-wmf.4/extensions/Flow/includes/Hooks.php(932): Flow\Formatter\ContributionsQuery->getResults(ContribsPager, string, integer, boolean)
#15 /srv/mediawiki/php-1.37.0-wmf.4/extensions/Flow/includes/Hooks.php(967): Flow\Hooks::onDeletedContributionsQuery(array, ContribsPager, string, integer, boolean)
#16 /srv/mediawiki/php-1.37.0-wmf.4/includes/HookContainer/HookContainer.php(330): Flow\Hooks::onContributionsQuery(array, ContribsPager, string, integer, boolean)
#17 /srv/mediawiki/php-1.37.0-wmf.4/includes/HookContainer/HookContainer.php(137): MediaWiki\HookContainer\HookContainer->callLegacyHook(string, array, array, array)
#18 /srv/mediawiki/php-1.37.0-wmf.4/includes/HookContainer/HookRunner.php(1300): MediaWiki\HookContainer\HookContainer->run(string, array)
#19 /srv/mediawiki/php-1.37.0-wmf.4/includes/specials/pagers/ContribsPager.php(262): MediaWiki\HookContainer\HookRunner->onContribsPager__reallyDoQuery(array, ContribsPager, string, integer, boolean)
#20 /srv/mediawiki/php-1.37.0-wmf.4/includes/pager/IndexPager.php(280): ContribsPager->reallyDoQuery(string, integer, boolean)
#21 /srv/mediawiki/php-1.37.0-wmf.4/includes/pager/IndexPager.php(733): IndexPager->doQuery()
#22 /srv/mediawiki/php-1.37.0-wmf.4/includes/specials/SpecialContributions.php(305): IndexPager->getNumRows()
#23 /srv/mediawiki/php-1.37.0-wmf.4/includes/specialpage/SpecialPage.php(646): SpecialContributions->execute(string)
#24 /srv/mediawiki/php-1.37.0-wmf.4/includes/specialpage/SpecialPageFactory.php(1396): SpecialPage->run(string)
#25 /srv/mediawiki/php-1.37.0-wmf.4/includes/MediaWiki.php(313): MediaWiki\SpecialPage\SpecialPageFactory->executePath(string, RequestContext)
#26 /srv/mediawiki/php-1.37.0-wmf.4/includes/MediaWiki.php(916): MediaWiki->performRequest()
#27 /srv/mediawiki/php-1.37.0-wmf.4/includes/MediaWiki.php(550): MediaWiki->main()
#28 /srv/mediawiki/php-1.37.0-wmf.4/index.php(53): MediaWiki->run()
#29 /srv/mediawiki/php-1.37.0-wmf.4/index.php(46): wfIndexMain()
#30 /srv/mediawiki/w/index.php(3): require(string)
#31 {main}
Impact
Notes

The same requests also report:

normalized_message
Expectation (readQueryTime <=) 5 by ApiMain::setRequestExpectations not met (actual: {actual}):
{query}
exception.trace
from /srv/mediawiki/php-1.37.0-wmf.4/includes/libs/rdbms/TransactionProfiler.php(432)
#0 /srv/mediawiki/php-1.37.0-wmf.4/includes/libs/rdbms/TransactionProfiler.php(272): Wikimedia\Rdbms\TransactionProfiler->reportExpectationViolated(string, Wikimedia\Rdbms\GeneralizedSql, double)
#1 /srv/mediawiki/php-1.37.0-wmf.4/includes/libs/rdbms/database/Database.php(1449): Wikimedia\Rdbms\TransactionProfiler->recordQueryCompletion(Wikimedia\Rdbms\GeneralizedSql, double, boolean, integer)
#2 /srv/mediawiki/php-1.37.0-wmf.4/includes/libs/rdbms/database/Database.php(1331): Wikimedia\Rdbms\Database->executeQueryAttempt(string, string, boolean, string, integer)
#3 /srv/mediawiki/php-1.37.0-wmf.4/includes/libs/rdbms/database/Database.php(1256): Wikimedia\Rdbms\Database->executeQuery(string, string, integer)
#4 /srv/mediawiki/php-1.37.0-wmf.4/includes/libs/rdbms/database/Database.php(1948): Wikimedia\Rdbms\Database->query(string, string, integer)
#5 /srv/mediawiki/php-1.37.0-wmf.4/includes/libs/rdbms/database/DBConnRef.php(68): Wikimedia\Rdbms\Database->select(string, array, array, string)
#6 /srv/mediawiki/php-1.37.0-wmf.4/includes/libs/rdbms/database/DBConnRef.php(313): Wikimedia\Rdbms\DBConnRef->__call(string, array)
#7 /srv/mediawiki/php-1.37.0-wmf.4/includes/externalstore/ExternalStoreDB.php(357): Wikimedia\Rdbms\DBConnRef->select(string, array, array, string)
#8 /srv/mediawiki/php-1.37.0-wmf.4/includes/externalstore/ExternalStoreDB.php(95): ExternalStoreDB->batchFetchBlobs(string, array)
#9 /srv/mediawiki/php-1.37.0-wmf.4/includes/externalstore/ExternalStoreAccess.php(70): ExternalStoreDB->batchFetchFromURLs(array)
#10 /srv/mediawiki/php-1.37.0-wmf.4/includes/externalstore/ExternalStore.php(120): ExternalStoreAccess->fetchFromURLs(array)
#11 /srv/mediawiki/php-1.37.0-wmf.4/extensions/Flow/includes/Data/Utils/Merger.php(79): ExternalStore::batchFetchFromURLs(array)
#12 /srv/mediawiki/php-1.37.0-wmf.4/extensions/Flow/includes/Data/Storage/RevisionStorage.php(350): Flow\Data\Utils\Merger::mergeMulti(array, string, array, string, string)
#13 /srv/mediawiki/php-1.37.0-wmf.4/extensions/Flow/includes/Data/Storage/RevisionStorage.php(174): Flow\Data\Storage\RevisionStorage::mergeExternalContent(array)
#14 /srv/mediawiki/php-1.37.0-wmf.4/extensions/Flow/includes/Data/Index/FeatureIndex.php(468): Flow\Data\Storage\RevisionStorage->findMulti(array, array)
#15 /srv/mediawiki/php-1.37.0-wmf.4/extensions/Flow/includes/Data/Index/FeatureIndex.php(282): Flow\Data\Index\FeatureIndex->backingStoreFindMulti(array)
#16 /srv/mediawiki/php-1.37.0-wmf.4/extensions/Flow/includes/Data/ObjectLocator.php(95): Flow\Data\Index\FeatureIndex->findMulti(array, array)
#17 /srv/mediawiki/php-1.37.0-wmf.4/extensions/Flow/includes/Data/ManagerGroup.php(127): Flow\Data\ObjectLocator->findMulti(array, array)
#18 /srv/mediawiki/php-1.37.0-wmf.4/extensions/Flow/includes/Data/ManagerGroup.php(143): Flow\Data\ManagerGroup->call(string, array)
#19 /srv/mediawiki/php-1.37.0-wmf.4/extensions/Flow/includes/Data/Listener/ReferenceRecorder.php(185): Flow\Data\ManagerGroup->findMulti(string, array, array)
#20 /srv/mediawiki/php-1.37.0-wmf.4/extensions/Flow/includes/Data/Listener/ReferenceRecorder.php(151): Flow\Data\Listener\ReferenceRecorder->collectTopicRevisions(Flow\Model\Workflow)
#21 /srv/mediawiki/php-1.37.0-wmf.4/extensions/Flow/includes/Data/Listener/ReferenceRecorder.php(80): Flow\Data\Listener\ReferenceRecorder->calculateChangesFromTopic(Flow\Model\Workflow, Flow\Model\PostRevision)
#22 /srv/mediawiki/php-1.37.0-wmf.4/extensions/Flow/includes/Data/ObjectManager.php(264): Flow\Data\Listener\ReferenceRecorder->onAfterInsert(Flow\Model\PostRevision, array, array)
#23 /srv/mediawiki/php-1.37.0-wmf.4/extensions/Flow/includes/Data/ObjectManager.php(173): Flow\Data\ObjectManager->insert(array, array)
#24 /srv/mediawiki/php-1.37.0-wmf.4/extensions/Flow/includes/Data/ObjectManager.php(146): Flow\Data\ObjectManager->multiPut(array, array)
#25 /srv/mediawiki/php-1.37.0-wmf.4/extensions/Flow/includes/Data/ManagerGroup.php(81): Flow\Data\ObjectManager->put(Flow\Model\PostRevision, array)
#26 /srv/mediawiki/php-1.37.0-wmf.4/extensions/Flow/includes/Block/TopicBlock.php(457): Flow\Data\ManagerGroup->put(Flow\Model\PostRevision, array)
#27 /srv/mediawiki/php-1.37.0-wmf.4/extensions/Flow/includes/SubmissionHandler.php(164): Flow\Block\TopicBlock->commit()
#28 /srv/mediawiki/php-1.37.0-wmf.4/extensions/Flow/includes/WorkflowLoader.php(66): Flow\SubmissionHandler->commit(Flow\Model\Workflow, array)
#29 /srv/mediawiki/php-1.37.0-wmf.4/extensions/Flow/includes/Api/ApiFlowBasePost.php(35): Flow\WorkflowLoader->commit(array)
#30 /srv/mediawiki/php-1.37.0-wmf.4/extensions/Flow/includes/Api/ApiFlow.php(108): Flow\Api\ApiFlowBasePost->execute()
#31 /srv/mediawiki/php-1.37.0-wmf.4/includes/api/ApiMain.php(1668): Flow\Api\ApiFlow->execute()
#32 /srv/mediawiki/php-1.37.0-wmf.4/includes/api/ApiMain.php(638): ApiMain->executeAction()
#33 /srv/mediawiki/php-1.37.0-wmf.4/includes/api/ApiMain.php(609): ApiMain->executeActionWithErrorHandling()
#34 /srv/mediawiki/php-1.37.0-wmf.4/api.php(90): ApiMain->execute()
#35 /srv/mediawiki/php-1.37.0-wmf.4/api.php(45): wfApiMain()
#36 /srv/mediawiki/w/api.php(3): require(string)
#37 {main}

Details

Request URL
https://www.wikidata.org/wiki/Special:Contributions/Quesotiotyo

Related Objects

Event Timeline

• mewoph created this task.May 11 2021, 6:51 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 11 2021, 6:51 PM
• mewoph moved this task from Inbox to Triaged on the Growth-Team board.May 11 2021, 7:08 PM
Krinkle renamed this task from Timed out waiting for replication to reach {wait_pos} to Special:Contributions with Flow entries throw "Timed out waiting for replication to reach" (via ExternalStoreDB).May 12 2021, 5:54 PM
Krinkle subscribed.May 12 2021, 6:02 PM

Does this happen regularly? And over the last 90 days, does it appear more now than before? This can be checked by searching for something like message:"Timed out waiting for replication" AND exception.trace:Flow on overall "mediawiki" Logstash dashboard.

If only rarely (e.g. handful of times a day or less), it might be "expected" if it correlates with operational events during which ExternalStoreDB's replication was reported to be lagged more generally. This can be checked via Grafana: MySQL aggregated.

Krinkle moved this task from Untriaged to May 2021 on the Wikimedia-production-error board.May 12 2021, 6:02 PM
Krinkle merged a task: T282601: Expectation (readQueryTime <=) 5 by ApiMain::setRequestExpectations not met (actual: {actual}): {query}.May 12 2021, 7:09 PM
Krinkle updated the task description. (Show Details)
Krinkle mentioned this in T281782: Flow: ContributionsFormatter times out for a certain user.Jun 12 2021, 6:24 PM
Krinkle mentioned this in Blog Post: Production Excellence #32: May 2021.Jun 21 2021, 1:31 AM
Etonkovidova subscribed.Jan 13 2022, 1:34 AM
In T282605#7083021, @Krinkle wrote:

Does this happen regularly? And over the last 90 days, does it appear more now than before? This can be checked by searching for something like message:"Timed out waiting for replication" AND exception.trace:Flow on overall "mediawiki" Logstash dashboard.

If only rarely (e.g. handful of times a day or less), it might be "expected" if it correlates with operational events during which ExternalStoreDB's replication was reported to be lagged more generally. This can be checked via Grafana: MySQL aggregated.

8 errors for message:"Timed out waiting for replication" AND exception.trace:Flow on wmf.17.

Krinkle moved this task from May 2021 to Older on the Wikimedia-production-error board.Jun 8 2023, 4:32 PM
Urbanecm_WMF removed a project: Growth-Team.Oct 28 2025, 12:27 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits