https://logstash.wikimedia.org/goto/622907ea32611577dfbaa80e335eed1c
We are having around 1,000 warnings every minute because of primary connection on GET requests. Most are coming from action=edit. This will slow down the system as it has to do cross dc queries.
| Subject | Repo | Branch | Lines +/- | |
|---|---|---|---|---|
| Read block data from replica in event logging code | mediawiki/extensions/WikimediaEvents | master | +1 -1 |
Almost 70% of the log entries in @Ladsgroup's search have this stack trace:
from /srv/mediawiki/php-1.43.0-wmf.7/includes/libs/rdbms/TransactionProfiler.php(525)
#0 /srv/mediawiki/php-1.43.0-wmf.7/includes/libs/rdbms/TransactionProfiler.php(265): Wikimedia\Rdbms\TransactionProfiler->reportExpectationViolated(string, string, integer)
#1 /srv/mediawiki/php-1.43.0-wmf.7/includes/libs/rdbms/loadbalancer/LoadBalancer.php(816): Wikimedia\Rdbms\TransactionProfiler->recordConnection(string, string, boolean)
#2 /srv/mediawiki/php-1.43.0-wmf.7/includes/libs/rdbms/loadbalancer/LoadBalancer.php(793): Wikimedia\Rdbms\LoadBalancer->getServerConnection(integer, string, integer)
#3 /srv/mediawiki/php-1.43.0-wmf.7/includes/libs/rdbms/database/DBConnRef.php(110): Wikimedia\Rdbms\LoadBalancer->getConnectionInternal(integer, array, string, integer)
#4 /srv/mediawiki/php-1.43.0-wmf.7/includes/libs/rdbms/database/DBConnRef.php(124): Wikimedia\Rdbms\DBConnRef->ensureConnection()
#5 /srv/mediawiki/php-1.43.0-wmf.7/includes/libs/rdbms/database/DBConnRef.php(424): Wikimedia\Rdbms\DBConnRef->__call(string, array)
#6 /srv/mediawiki/php-1.43.0-wmf.7/includes/block/DatabaseBlockStore.php(407): Wikimedia\Rdbms\DBConnRef->makeList(array, integer)
#7 /srv/mediawiki/php-1.43.0-wmf.7/includes/block/DatabaseBlockStore.php(695): MediaWiki\Block\DatabaseBlockStore->newLoad(MediaWiki\User\User, integer, boolean, string)
#8 /srv/mediawiki/php-1.43.0-wmf.7/includes/block/BlockManager.php(230): MediaWiki\Block\DatabaseBlockStore->newListFromTarget(MediaWiki\User\User, string, boolean)
#9 /srv/mediawiki/php-1.43.0-wmf.7/includes/Permissions/PermissionManager.php(898): MediaWiki\Block\BlockManager->getBlock(MediaWiki\User\User, MediaWiki\Request\WebRequest, boolean)
#10 /srv/mediawiki/php-1.43.0-wmf.7/includes/Permissions/PermissionManager.php(444): MediaWiki\Permissions\PermissionManager->getApplicableBlock(string, MediaWiki\User\User, string, MediaWiki\Title\Title, MediaWiki\Request\WebRequest)
#11 /srv/mediawiki/php-1.43.0-wmf.7/extensions/WikimediaEvents/includes/BlockUtils.php(73): MediaWiki\Permissions\PermissionManager->isBlockedFrom(MediaWiki\User\User, MediaWiki\Title\Title, boolean)
#12 /srv/mediawiki/php-1.43.0-wmf.7/extensions/WikimediaEvents/includes/EditPage/EditPageHooks.php(45): WikimediaEvents\BlockUtils::logBlockedEditAttempt(MediaWiki\User\User, MediaWiki\Title\Title, string, string)
#13 /srv/mediawiki/php-1.43.0-wmf.7/includes/HookContainer/HookContainer.php(159): WikimediaEvents\EditPage\EditPageHooks->onEditPage__showReadOnlyForm_initial(MediaWiki\EditPage\EditPage, MediaWiki\Output\OutputPage)
#14 /srv/mediawiki/php-1.43.0-wmf.7/includes/HookContainer/HookRunner.php(1597): MediaWiki\HookContainer\HookContainer->run(string, array)
#15 /srv/mediawiki/php-1.43.0-wmf.7/includes/editpage/EditPage.php(976): MediaWiki\HookContainer\HookRunner->onEditPage__showReadOnlyForm_initial(MediaWiki\EditPage\EditPage, MediaWiki\Output\OutputPage)
#16 /srv/mediawiki/php-1.43.0-wmf.7/includes/editpage/EditPage.php(965): MediaWiki\EditPage\EditPage->displayViewSourcePage(WikitextContent, string)
#17 /srv/mediawiki/php-1.43.0-wmf.7/includes/editpage/EditPage.php(665): MediaWiki\EditPage\EditPage->displayPermissionsError(array)
#18 /srv/mediawiki/php-1.43.0-wmf.7/includes/actions/EditAction.php(66): MediaWiki\EditPage\EditPage->edit()
#19 /srv/mediawiki/php-1.43.0-wmf.7/includes/actions/ActionEntryPoint.php(731): EditAction->show()
#20 /srv/mediawiki/php-1.43.0-wmf.7/includes/actions/ActionEntryPoint.php(508): MediaWiki\Actions\ActionEntryPoint->performAction(Article, MediaWiki\Title\Title)
#21 /srv/mediawiki/php-1.43.0-wmf.7/includes/actions/ActionEntryPoint.php(145): MediaWiki\Actions\ActionEntryPoint->performRequest()
#22 /srv/mediawiki/php-1.43.0-wmf.7/includes/MediaWikiEntryPoint.php(200): MediaWiki\Actions\ActionEntryPoint->execute()
#23 /srv/mediawiki/php-1.43.0-wmf.7/index.php(58): MediaWiki\MediaWikiEntryPoint->run()
#24 /srv/mediawiki/w/index.php(3): require(string)
#25 {main}That code was introduced in https://gerrit.wikimedia.org/r/c/mediawiki/extensions/WikimediaEvents/+/832148 almost 2 years ago, so this doesn't seem terribly urgent. It looks like an easy fix though.
Change #1039262 had a related patch set uploaded (by Bartosz Dziewoński; author: Bartosz Dziewoński):
[mediawiki/extensions/WikimediaEvents@master] Read block data from replica in event logging code
Change #1039262 merged by jenkins-bot:
[mediawiki/extensions/WikimediaEvents@master] Read block data from replica in event logging code
The others are much lower volume, and at a glance, there were many different stack traces coming from many different components, including some semi-abandoned ones like LiquidThreads and FlaggedRevs, so presumably they have been there for not just 2 years but 10+ years. You can look through the stack traces, file tasks and write patches for each one, but it doesn't seem worth the effort to me.