Page MenuHomePhabricator
Create Task
Maniphest T374534

Lots of "Expectation (masterConns <= 0) by ApiMain::setRequestExpectations not met" involving external store (2024-09-05)
Closed, ResolvedPublic
Actions

Assigned To
aaron
Authored By
• matmarex
Sep 11 2024, 12:49 PM
Project Tags
Referenced Files
F57500816: grafik.png
Sep 11 2024, 11:06 PM
F57499497: image.png
Sep 11 2024, 12:49 PM
F57499493: image.png
Sep 11 2024, 12:49 PM
Subscribers
aaron
Aklapper
BPirkle
Ladsgroup
• matmarex
Umherirrender
Zabe

Description

I was looking at some dashboards (classic mistake) and noticed that on 2024-09-05, the volume of rdbms WARNING logs, already high on a good day, has increased by about 20 times.
https://logstash.wikimedia.org/goto/184d12a2dfda698ecd364078a8198daf

image.png (2,780×536 px, 85 KB)

Most of the increase can be attributed to warnings like "Expectation (masterConns <= 0) by ApiMain::setRequestExpectations not met" with ExternalStoreDB in the stack trace.
https://logstash.wikimedia.org/goto/c0ca043c67ccadda3cdc0d4d2be249ff

image.png (2,780×532 px, 83 KB)

They happen in a variety of API requests that fetch revisions from the external store. Some example stack traces:

from /srv/mediawiki/php-1.43.0-wmf.21/includes/libs/rdbms/TransactionProfiler.php(526)
#0 /srv/mediawiki/php-1.43.0-wmf.21/includes/libs/rdbms/TransactionProfiler.php(265): Wikimedia\Rdbms\TransactionProfiler->reportExpectationViolated(string, string, int)
#1 /srv/mediawiki/php-1.43.0-wmf.21/includes/libs/rdbms/loadbalancer/LoadBalancer.php(804): Wikimedia\Rdbms\TransactionProfiler->recordConnection(string, string, bool)
#2 /srv/mediawiki/php-1.43.0-wmf.21/includes/libs/rdbms/loadbalancer/LoadBalancer.php(781): Wikimedia\Rdbms\LoadBalancer->getServerConnection(int, string, int)
#3 /srv/mediawiki/php-1.43.0-wmf.21/includes/libs/rdbms/database/DBConnRef.php(111): Wikimedia\Rdbms\LoadBalancer->getConnectionInternal(int, array, string, int)
#4 /srv/mediawiki/php-1.43.0-wmf.21/includes/libs/rdbms/database/DBConnRef.php(125): Wikimedia\Rdbms\DBConnRef->ensureConnection()
#5 /srv/mediawiki/php-1.43.0-wmf.21/includes/libs/rdbms/database/DBConnRef.php(338): Wikimedia\Rdbms\DBConnRef->__call(string, array)
#6 /srv/mediawiki/php-1.43.0-wmf.21/includes/libs/rdbms/querybuilder/SelectQueryBuilder.php(764): Wikimedia\Rdbms\DBConnRef->selectField(array, string, array, string, array, array)
#7 /srv/mediawiki/php-1.43.0-wmf.21/includes/externalstore/ExternalStoreDB.php(312): Wikimedia\Rdbms\SelectQueryBuilder->fetchField()
#8 /srv/mediawiki/php-1.43.0-wmf.21/includes/externalstore/ExternalStoreDB.php(67): ExternalStoreDB->fetchBlob(string, string, bool)
#9 /srv/mediawiki/php-1.43.0-wmf.21/includes/externalstore/ExternalStoreAccess.php(73): ExternalStoreDB->fetchFromURL(string)
#10 /srv/mediawiki/php-1.43.0-wmf.21/includes/Storage/SqlBlobStore.php(571): ExternalStoreAccess->fetchFromURL(string, array)
#11 /srv/mediawiki/php-1.43.0-wmf.21/includes/libs/objectcache/wancache/WANObjectCache.php(1729): MediaWiki\Storage\SqlBlobStore->MediaWiki\Storage\{closure}(bool, int, array, null, array)
#12 /srv/mediawiki/php-1.43.0-wmf.21/includes/libs/objectcache/wancache/WANObjectCache.php(1559): WANObjectCache->fetchOrRegenerate(string, int, Closure, array, array)
#13 /srv/mediawiki/php-1.43.0-wmf.21/includes/Storage/SqlBlobStore.php(575): WANObjectCache->getWithSetCallback(string, int, Closure, array)
#14 /srv/mediawiki/php-1.43.0-wmf.21/includes/Storage/SqlBlobStore.php(438): MediaWiki\Storage\SqlBlobStore->expandBlob(string, array, string)
#15 /srv/mediawiki/php-1.43.0-wmf.21/includes/Storage/SqlBlobStore.php(264): MediaWiki\Storage\SqlBlobStore->fetchBlobs(array, int)
#16 /srv/mediawiki/php-1.43.0-wmf.21/includes/libs/objectcache/wancache/WANObjectCache.php(1729): MediaWiki\Storage\SqlBlobStore->MediaWiki\Storage\{closure}(bool, int, array, null, array)
#17 /srv/mediawiki/php-1.43.0-wmf.21/includes/libs/objectcache/wancache/WANObjectCache.php(1559): WANObjectCache->fetchOrRegenerate(string, int, Closure, array, array)
#18 /srv/mediawiki/php-1.43.0-wmf.21/includes/Storage/SqlBlobStore.php(272): WANObjectCache->getWithSetCallback(string, int, Closure, array)
#19 /srv/mediawiki/php-1.43.0-wmf.21/includes/Revision/RevisionStore.php(1208): MediaWiki\Storage\SqlBlobStore->getBlob(string, int)
#20 /srv/mediawiki/php-1.43.0-wmf.21/includes/Revision/RevisionStore.php(1519): MediaWiki\Revision\RevisionStore->loadSlotContent(MediaWiki\Revision\SlotRecord, null, null, null, int)
#21 [internal function]: MediaWiki\Revision\RevisionStore->MediaWiki\Revision\{closure}(MediaWiki\Revision\SlotRecord)
#22 /srv/mediawiki/php-1.43.0-wmf.21/includes/Revision/SlotRecord.php(324): call_user_func(Closure, MediaWiki\Revision\SlotRecord)
#23 /srv/mediawiki/php-1.43.0-wmf.21/includes/Revision/RevisionRecord.php(158): MediaWiki\Revision\SlotRecord->getContent()
#24 /srv/mediawiki/php-1.43.0-wmf.21/includes/page/WikiPage.php(770): MediaWiki\Revision\RevisionRecord->getContent(string, int, null)
#25 /srv/mediawiki/php-1.43.0-wmf.21/extensions/CirrusSearch/includes/Sanity/Checker.php(244): WikiPage->getContent()
#26 /srv/mediawiki/php-1.43.0-wmf.21/extensions/CirrusSearch/includes/Sanity/Checker.php(216): CirrusSearch\Sanity\Checker->checkIfRedirect(WikiPage)
#27 /srv/mediawiki/php-1.43.0-wmf.21/extensions/CirrusSearch/includes/Sanity/Checker.php(172): CirrusSearch\Sanity\Checker->checkExisitingPage(string, int, WikiPage, array)
#28 /srv/mediawiki/php-1.43.0-wmf.21/extensions/CirrusSearch/includes/Api/CheckSanity.php(104): CirrusSearch\Sanity\Checker->check(array)
#29 /srv/mediawiki/php-1.43.0-wmf.21/extensions/CirrusSearch/includes/Api/CheckSanity.php(60): CirrusSearch\Api\CheckSanity->check(CirrusSearch\Sanity\Checker, int, int)
#30 /srv/mediawiki/php-1.43.0-wmf.21/includes/api/ApiMain.php(1948): CirrusSearch\Api\CheckSanity->execute()
#31 /srv/mediawiki/php-1.43.0-wmf.21/includes/api/ApiMain.php(926): ApiMain->executeAction()
#32 /srv/mediawiki/php-1.43.0-wmf.21/includes/api/ApiMain.php(897): ApiMain->executeActionWithErrorHandling()
#33 /srv/mediawiki/php-1.43.0-wmf.21/includes/api/ApiEntryPoint.php(153): ApiMain->execute()
#34 /srv/mediawiki/php-1.43.0-wmf.21/includes/MediaWikiEntryPoint.php(200): MediaWiki\Api\ApiEntryPoint->execute()
#35 /srv/mediawiki/php-1.43.0-wmf.21/api.php(44): MediaWiki\MediaWikiEntryPoint->run()
#36 /srv/mediawiki/w/api.php(3): require(string)
#37 {main}
from /srv/mediawiki/php-1.43.0-wmf.21/includes/libs/rdbms/TransactionProfiler.php(526)
#0 /srv/mediawiki/php-1.43.0-wmf.21/includes/libs/rdbms/TransactionProfiler.php(265): Wikimedia\Rdbms\TransactionProfiler->reportExpectationViolated(string, string, int)
#1 /srv/mediawiki/php-1.43.0-wmf.21/includes/libs/rdbms/loadbalancer/LoadBalancer.php(804): Wikimedia\Rdbms\TransactionProfiler->recordConnection(string, string, bool)
#2 /srv/mediawiki/php-1.43.0-wmf.21/includes/libs/rdbms/loadbalancer/LoadBalancer.php(781): Wikimedia\Rdbms\LoadBalancer->getServerConnection(int, string, int)
#3 /srv/mediawiki/php-1.43.0-wmf.21/includes/libs/rdbms/database/DBConnRef.php(111): Wikimedia\Rdbms\LoadBalancer->getConnectionInternal(int, array, string, int)
#4 /srv/mediawiki/php-1.43.0-wmf.21/includes/libs/rdbms/database/DBConnRef.php(125): Wikimedia\Rdbms\DBConnRef->ensureConnection()
#5 /srv/mediawiki/php-1.43.0-wmf.21/includes/libs/rdbms/database/DBConnRef.php(338): Wikimedia\Rdbms\DBConnRef->__call(string, array)
#6 /srv/mediawiki/php-1.43.0-wmf.21/includes/libs/rdbms/querybuilder/SelectQueryBuilder.php(764): Wikimedia\Rdbms\DBConnRef->selectField(array, string, array, string, array, array)
#7 /srv/mediawiki/php-1.43.0-wmf.21/includes/externalstore/ExternalStoreDB.php(312): Wikimedia\Rdbms\SelectQueryBuilder->fetchField()
#8 /srv/mediawiki/php-1.43.0-wmf.21/includes/externalstore/ExternalStoreDB.php(67): ExternalStoreDB->fetchBlob(string, string, bool)
#9 /srv/mediawiki/php-1.43.0-wmf.21/includes/externalstore/ExternalStoreAccess.php(73): ExternalStoreDB->fetchFromURL(string)
#10 /srv/mediawiki/php-1.43.0-wmf.21/includes/Storage/SqlBlobStore.php(571): ExternalStoreAccess->fetchFromURL(string, array)
#11 /srv/mediawiki/php-1.43.0-wmf.21/includes/libs/objectcache/wancache/WANObjectCache.php(1729): MediaWiki\Storage\SqlBlobStore->MediaWiki\Storage\{closure}(bool, int, array, null, array)
#12 /srv/mediawiki/php-1.43.0-wmf.21/includes/libs/objectcache/wancache/WANObjectCache.php(1559): WANObjectCache->fetchOrRegenerate(string, int, Closure, array, array)
#13 /srv/mediawiki/php-1.43.0-wmf.21/includes/Storage/SqlBlobStore.php(575): WANObjectCache->getWithSetCallback(string, int, Closure, array)
#14 /srv/mediawiki/php-1.43.0-wmf.21/includes/Storage/SqlBlobStore.php(438): MediaWiki\Storage\SqlBlobStore->expandBlob(string, array, string)
#15 /srv/mediawiki/php-1.43.0-wmf.21/includes/Storage/SqlBlobStore.php(264): MediaWiki\Storage\SqlBlobStore->fetchBlobs(array, int)
#16 /srv/mediawiki/php-1.43.0-wmf.21/includes/libs/objectcache/wancache/WANObjectCache.php(1729): MediaWiki\Storage\SqlBlobStore->MediaWiki\Storage\{closure}(bool, int, array, null, array)
#17 /srv/mediawiki/php-1.43.0-wmf.21/includes/libs/objectcache/wancache/WANObjectCache.php(1559): WANObjectCache->fetchOrRegenerate(string, int, Closure, array, array)
#18 /srv/mediawiki/php-1.43.0-wmf.21/includes/Storage/SqlBlobStore.php(272): WANObjectCache->getWithSetCallback(string, int, Closure, array)
#19 /srv/mediawiki/php-1.43.0-wmf.21/includes/Revision/RevisionStore.php(1208): MediaWiki\Storage\SqlBlobStore->getBlob(string, int)
#20 /srv/mediawiki/php-1.43.0-wmf.21/includes/Revision/RevisionStore.php(1519): MediaWiki\Revision\RevisionStore->loadSlotContent(MediaWiki\Revision\SlotRecord, null, null, null, int)
#21 [internal function]: MediaWiki\Revision\RevisionStore->MediaWiki\Revision\{closure}(MediaWiki\Revision\SlotRecord)
#22 /srv/mediawiki/php-1.43.0-wmf.21/includes/Revision/SlotRecord.php(324): call_user_func(Closure, MediaWiki\Revision\SlotRecord)
#23 /srv/mediawiki/php-1.43.0-wmf.21/includes/api/ApiQueryRevisionsBase.php(605): MediaWiki\Revision\SlotRecord->getContent()
#24 /srv/mediawiki/php-1.43.0-wmf.21/includes/api/ApiQueryRevisionsBase.php(532): ApiQueryRevisionsBase->extractSlotInfo(MediaWiki\Revision\SlotRecord, int, null)
#25 /srv/mediawiki/php-1.43.0-wmf.21/includes/api/ApiQueryRevisionsBase.php(426): ApiQueryRevisionsBase->extractAllSlotInfo(MediaWiki\Revision\RevisionStoreRecord, int)
#26 /srv/mediawiki/php-1.43.0-wmf.21/includes/api/ApiQueryRevisions.php(439): ApiQueryRevisionsBase->extractRevisionInfo(MediaWiki\Revision\RevisionStoreRecord, stdClass)
#27 /srv/mediawiki/php-1.43.0-wmf.21/includes/api/ApiQueryRevisionsBase.php(166): ApiQueryRevisions->run()
#28 /srv/mediawiki/php-1.43.0-wmf.21/includes/api/ApiQuery.php(711): ApiQueryRevisionsBase->execute()
#29 /srv/mediawiki/php-1.43.0-wmf.21/includes/api/ApiMain.php(1948): ApiQuery->execute()
#30 /srv/mediawiki/php-1.43.0-wmf.21/includes/api/ApiMain.php(926): ApiMain->executeAction()
#31 /srv/mediawiki/php-1.43.0-wmf.21/includes/api/ApiMain.php(897): ApiMain->executeActionWithErrorHandling()
#32 /srv/mediawiki/php-1.43.0-wmf.21/includes/api/ApiEntryPoint.php(153): ApiMain->execute()
#33 /srv/mediawiki/php-1.43.0-wmf.21/includes/MediaWikiEntryPoint.php(200): MediaWiki\Api\ApiEntryPoint->execute()
#34 /srv/mediawiki/php-1.43.0-wmf.21/api.php(44): MediaWiki\MediaWikiEntryPoint->run()
#35 /srv/mediawiki/w/api.php(3): require(string)
#36 {main}
from /srv/mediawiki/php-1.43.0-wmf.21/includes/libs/rdbms/TransactionProfiler.php(526)
#0 /srv/mediawiki/php-1.43.0-wmf.21/includes/libs/rdbms/TransactionProfiler.php(265): Wikimedia\Rdbms\TransactionProfiler->reportExpectationViolated(string, string, int)
#1 /srv/mediawiki/php-1.43.0-wmf.21/includes/libs/rdbms/loadbalancer/LoadBalancer.php(804): Wikimedia\Rdbms\TransactionProfiler->recordConnection(string, string, bool)
#2 /srv/mediawiki/php-1.43.0-wmf.21/includes/libs/rdbms/loadbalancer/LoadBalancer.php(781): Wikimedia\Rdbms\LoadBalancer->getServerConnection(int, string, int)
#3 /srv/mediawiki/php-1.43.0-wmf.21/includes/libs/rdbms/database/DBConnRef.php(111): Wikimedia\Rdbms\LoadBalancer->getConnectionInternal(int, array, string, int)
#4 /srv/mediawiki/php-1.43.0-wmf.21/includes/libs/rdbms/database/DBConnRef.php(125): Wikimedia\Rdbms\DBConnRef->ensureConnection()
#5 /srv/mediawiki/php-1.43.0-wmf.21/includes/libs/rdbms/database/DBConnRef.php(338): Wikimedia\Rdbms\DBConnRef->__call(string, array)
#6 /srv/mediawiki/php-1.43.0-wmf.21/includes/libs/rdbms/querybuilder/SelectQueryBuilder.php(764): Wikimedia\Rdbms\DBConnRef->selectField(array, string, array, string, array, array)
#7 /srv/mediawiki/php-1.43.0-wmf.21/includes/externalstore/ExternalStoreDB.php(312): Wikimedia\Rdbms\SelectQueryBuilder->fetchField()
#8 /srv/mediawiki/php-1.43.0-wmf.21/includes/externalstore/ExternalStoreDB.php(67): ExternalStoreDB->fetchBlob(string, string, bool)
#9 /srv/mediawiki/php-1.43.0-wmf.21/includes/externalstore/ExternalStoreAccess.php(73): ExternalStoreDB->fetchFromURL(string)
#10 /srv/mediawiki/php-1.43.0-wmf.21/includes/Storage/SqlBlobStore.php(571): ExternalStoreAccess->fetchFromURL(string, array)
#11 /srv/mediawiki/php-1.43.0-wmf.21/includes/libs/objectcache/wancache/WANObjectCache.php(1729): MediaWiki\Storage\SqlBlobStore->MediaWiki\Storage\{closure}(bool, int, array, null, array)
#12 /srv/mediawiki/php-1.43.0-wmf.21/includes/libs/objectcache/wancache/WANObjectCache.php(1559): WANObjectCache->fetchOrRegenerate(string, int, Closure, array, array)
#13 /srv/mediawiki/php-1.43.0-wmf.21/includes/Storage/SqlBlobStore.php(575): WANObjectCache->getWithSetCallback(string, int, Closure, array)
#14 /srv/mediawiki/php-1.43.0-wmf.21/includes/Storage/SqlBlobStore.php(438): MediaWiki\Storage\SqlBlobStore->expandBlob(string, array, string)
#15 /srv/mediawiki/php-1.43.0-wmf.21/includes/Storage/SqlBlobStore.php(264): MediaWiki\Storage\SqlBlobStore->fetchBlobs(array, int)
#16 /srv/mediawiki/php-1.43.0-wmf.21/includes/libs/objectcache/wancache/WANObjectCache.php(1729): MediaWiki\Storage\SqlBlobStore->MediaWiki\Storage\{closure}(bool, int, array, null, array)
#17 /srv/mediawiki/php-1.43.0-wmf.21/includes/libs/objectcache/wancache/WANObjectCache.php(1559): WANObjectCache->fetchOrRegenerate(string, int, Closure, array, array)
#18 /srv/mediawiki/php-1.43.0-wmf.21/includes/Storage/SqlBlobStore.php(272): WANObjectCache->getWithSetCallback(string, int, Closure, array)
#19 /srv/mediawiki/php-1.43.0-wmf.21/includes/Revision/RevisionStore.php(1208): MediaWiki\Storage\SqlBlobStore->getBlob(string, int)
#20 /srv/mediawiki/php-1.43.0-wmf.21/includes/Revision/RevisionStore.php(1519): MediaWiki\Revision\RevisionStore->loadSlotContent(MediaWiki\Revision\SlotRecord, null, null, null, int)
#21 [internal function]: MediaWiki\Revision\RevisionStore->MediaWiki\Revision\{closure}(MediaWiki\Revision\SlotRecord)
#22 /srv/mediawiki/php-1.43.0-wmf.21/includes/Revision/SlotRecord.php(324): call_user_func(Closure, MediaWiki\Revision\SlotRecord)
#23 /srv/mediawiki/php-1.43.0-wmf.21/includes/Revision/RevisionRecord.php(183): MediaWiki\Revision\SlotRecord->getContent()
#24 /srv/mediawiki/php-1.43.0-wmf.21/includes/Revision/RenderedRevision.php(231): MediaWiki\Revision\RevisionRecord->getContentOrThrow(string, int, null)
#25 /srv/mediawiki/php-1.43.0-wmf.21/includes/Revision/RevisionRenderer.php(234): MediaWiki\Revision\RenderedRevision->getSlotParserOutput(string, array)
#26 /srv/mediawiki/php-1.43.0-wmf.21/includes/Revision/RevisionRenderer.php(170): MediaWiki\Revision\RevisionRenderer->combineSlotOutput(MediaWiki\Revision\RenderedRevision, ParserOptions, array)
#27 [internal function]: MediaWiki\Revision\RevisionRenderer->MediaWiki\Revision\{closure}(MediaWiki\Revision\RenderedRevision, array)
#28 /srv/mediawiki/php-1.43.0-wmf.21/includes/Revision/RenderedRevision.php(199): call_user_func(Closure, MediaWiki\Revision\RenderedRevision, array)
#29 /srv/mediawiki/php-1.43.0-wmf.21/includes/page/ParserOutputAccess.php(443): MediaWiki\Revision\RenderedRevision->getRevisionParserOutput()
#30 /srv/mediawiki/php-1.43.0-wmf.21/includes/page/ParserOutputAccess.php(362): MediaWiki\Page\ParserOutputAccess->renderRevision(WikiPage, ParserOptions, MediaWiki\Revision\RevisionStoreRecord, int, null)
#31 /srv/mediawiki/php-1.43.0-wmf.21/includes/page/WikiPage.php(1126): MediaWiki\Page\ParserOutputAccess->getParserOutput(WikiPage, ParserOptions, MediaWiki\Revision\RevisionStoreRecord, int)
#32 /srv/mediawiki/php-1.43.0-wmf.21/includes/api/ApiParse.php(194): WikiPage->getParserOutput(ParserOptions, null, bool)
#33 /srv/mediawiki/php-1.43.0-wmf.21/includes/poolcounter/PoolCounterWorkViaCallback.php(73): ApiParse::{closure}()
#34 /srv/mediawiki/php-1.43.0-wmf.21/includes/poolcounter/PoolCounterWork.php(171): MediaWiki\PoolCounter\PoolCounterWorkViaCallback->doWork()
#35 /srv/mediawiki/php-1.43.0-wmf.21/includes/api/ApiParse.php(201): MediaWiki\PoolCounter\PoolCounterWork->execute()
#36 /srv/mediawiki/php-1.43.0-wmf.21/includes/api/ApiParse.php(859): ApiParse->getPageParserOutput(WikiPage, null, ParserOptions, bool)
#37 /srv/mediawiki/php-1.43.0-wmf.21/includes/api/ApiParse.php(326): ApiParse->getParsedContent(WikiPage, ParserOptions, bool, null, null, bool)
#38 /srv/mediawiki/php-1.43.0-wmf.21/includes/api/ApiMain.php(1948): ApiParse->execute()
#39 /srv/mediawiki/php-1.43.0-wmf.21/includes/api/ApiMain.php(926): ApiMain->executeAction()
#40 /srv/mediawiki/php-1.43.0-wmf.21/includes/api/ApiMain.php(897): ApiMain->executeActionWithErrorHandling()
#41 /srv/mediawiki/php-1.43.0-wmf.21/includes/api/ApiEntryPoint.php(153): ApiMain->execute()
#42 /srv/mediawiki/php-1.43.0-wmf.21/includes/MediaWikiEntryPoint.php(200): MediaWiki\Api\ApiEntryPoint->execute()
#43 /srv/mediawiki/php-1.43.0-wmf.21/api.php(44): MediaWiki\MediaWikiEntryPoint->run()
#44 /srv/mediawiki/w/api.php(3): require(string)
#45 {main}

Details

Related Changes in Gerrit:
SubjectAuthorRepoBranchLines +/-
externalstore: disable DB_PRIMARY fallback for static data clustersAaron Schulzmediawiki/coremaster+13 -2
rdbms: only count replication sources toward "masterConns" in getServerConnection()Hasharmediawiki/corewmf/1.43.0-wmf.22+2 -2
rdbms: only count replication sources toward "masterConns" in getServerConnection()Aaron Schulzmediawiki/coremaster+2 -2
Customize query in gerrit

Related Objects

Event Timeline

• matmarex created this task.Sep 11 2024, 12:49 PM
Herald added a subscriber: Aklapper. · View Herald TranscriptSep 11 2024, 12:49 PM
Umherirrender added subscribers: Zabe, Ladsgroup, Umherirrender.Sep 11 2024, 7:11 PM

line 312 refers to the replica read https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/core/+/refs/heads/wmf/1.43.0-wmf.21/includes/externalstore/ExternalStoreDB.php#312

In case of a primary db fallback the warning __METHOD__ . ": primary DB fallback on $cacheID" is logged (from ExternalStoreDB::fetchBlob), I would not expect that here.

ExternalStoreDB::getReplica is using DB_REPLICA

There is some migration to external storage in T183490: MCR schema migration stage 4: Migrate External Store URLs (wmf production), maybe related that there are more logs (but not the root cause)

Ladsgroup added a comment.Sep 11 2024, 8:18 PM

It can't be related to text table clean up: the timing doesn't match. CLI can make connections to master. This is GET requests. I take a look soon.

Umherirrender added a comment.Sep 11 2024, 9:04 PM

ccf94e31c134bdbc11ee5dd668edb016eea9cf4a could be related, it changed the (meaning of the) condition for TransactionProfiler issue type masterConns

The use of CONN_TRX_AUTOCOMMIT for the replica seems not to make trouble, but could be without any effect, only useful when writing to the database (from my point of view).

Ladsgroup added a comment.Sep 11 2024, 10:58 PM

It is exactly what I feared it to be. es1 to es5 are RO (no replication set between the hosts and writes would fail). They have a master set in the config but it's not really a master and that's why it has the exact same weight as other hosts in that section. Somewhere the check for hasReplicaServers() needs to change to hasStreamingReplicaServers()

Ladsgroup added a project: MediaWiki-API-Platform-Team.Sep 11 2024, 11:06 PM
Ladsgroup added subscribers: BPirkle, aaron.

Looking at the host makes the problem quite clear:

grafik.png (185×865 px, 45 KB)

(all these hosts are set as "master" in RO sections)

This is quite important to fix. The flood of logspam is quite large and is drowning important warnings.

Krinkle moved this task from Untriaged to Sep 2024 on the Wikimedia-production-error board.Sep 12 2024, 12:35 AM
BPirkle changed the task status from Open to In Progress.Sep 12 2024, 3:22 PM
BPirkle assigned this task to aaron.
BPirkle triaged this task as High priority.
BPirkle moved this task from Incoming (Needs Triage) to Backlog on the MediaWiki-API-Platform-Team board.
BPirkle moved this task from Backlog to In Progress on the MediaWiki-API-Platform-Team board.
aaron added a comment.Sep 12 2024, 4:35 PM

Isn't the fallback useless in this case, since there won't be anything on the primary that is not on the replica? Also, why is some non-existing blob being referenced so much? From etcd.php in mediawiki-config, it looks like DB_PRIMARY will still use a server in the primary datacenter, even for 'static' clusters, meaning that it's still a cross-DC query sometimes. This makes it seem like the warning is legitimate.

I'm not really fond of replica->primary fallback logic for this reason, as issues like this tend to crop up.

aaron added a comment.Sep 12 2024, 9:57 PM

The logging code should check hasStreamingReplicaServers(), though it will then incidentally ignore cross-DC queries to static clones servers.

gerritbot added a comment.Sep 13 2024, 7:38 AM

Change #1072694 had a related patch set uploaded (by Aaron Schulz; author: Aaron Schulz):

[mediawiki/core@master] rdbms: only count replication sources toward "masterConns" in getServerConnection()

https://gerrit.wikimedia.org/r/1072694

gerritbot added a project: Patch-For-Review.Sep 13 2024, 7:38 AM
Ladsgroup added a comment.EditedSep 13 2024, 11:16 AM
In T374534#10141679, @aaron wrote:

Isn't the fallback useless in this case, since there won't be anything on the primary that is not on the replica? Also, why is some non-existing blob being referenced so much? From etcd.php in mediawiki-config, it looks like DB_PRIMARY will still use a server in the primary datacenter, even for 'static' clusters, meaning that it's still a cross-DC query sometimes. This makes it seem like the warning is legitimate.

FWIW, if you search for warnings from codfw, only 100 warnings per six hours shows up. All of these warnings are from eqiad which is the primary dc and thus no cross-dc query happening.

gerritbot added a comment.Sep 13 2024, 11:48 AM

Change #1072694 merged by jenkins-bot:

[mediawiki/core@master] rdbms: only count replication sources toward "masterConns" in getServerConnection()

https://gerrit.wikimedia.org/r/1072694

ReleaseTaggerBot added a project: MW-1.43-notes (1.43.0-wmf.23; 2024-09-17).Sep 13 2024, 12:00 PM
Maintenance_bot removed a project: Patch-For-Review.Sep 13 2024, 12:30 PM
gerritbot added a comment.Sep 13 2024, 12:43 PM

Change #1072739 had a related patch set uploaded (by Hashar; author: Aaron Schulz):

[mediawiki/core@wmf/1.43.0-wmf.22] rdbms: only count replication sources toward "masterConns" in getServerConnection()

https://gerrit.wikimedia.org/r/1072739

gerritbot added a project: Patch-For-Review.Sep 13 2024, 12:43 PM
gerritbot added a comment.Sep 13 2024, 1:15 PM

Change #1072739 merged by jenkins-bot:

[mediawiki/core@wmf/1.43.0-wmf.22] rdbms: only count replication sources toward "masterConns" in getServerConnection()

https://gerrit.wikimedia.org/r/1072739

Stashbot added a comment.Sep 13 2024, 1:16 PM

Mentioned in SAL (#wikimedia-operations) [2024-09-13T13:16:53Z] <hashar@deploy1003> Started scap sync-world: Backport for [[gerrit:1072739|rdbms: only count replication sources toward "masterConns" in getServerConnection() (T374534)]]

Stashbot added a comment.Sep 13 2024, 1:20 PM

Mentioned in SAL (#wikimedia-operations) [2024-09-13T13:20:10Z] <hashar@deploy1003> hashar: Backport for [[gerrit:1072739|rdbms: only count replication sources toward "masterConns" in getServerConnection() (T374534)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)

Stashbot added a comment.Sep 13 2024, 1:27 PM

Mentioned in SAL (#wikimedia-operations) [2024-09-13T13:27:28Z] <hashar@deploy1003> Finished scap sync-world: Backport for [[gerrit:1072739|rdbms: only count replication sources toward "masterConns" in getServerConnection() (T374534)]] (duration: 10m 34s)

Maintenance_bot removed a project: Patch-For-Review.Sep 13 2024, 1:31 PM
ReleaseTaggerBot edited projects, added: MW-1.43-notes (1.43.0-wmf.22; 2024-09-10); removed: MW-1.43-notes (1.43.0-wmf.23; 2024-09-17).Sep 13 2024, 2:00 PM
aaron added a comment.EditedSep 13 2024, 10:56 PM

Even without the fallback logic, the setup seems odd. Since the es5 cluster config has 100 weight for the writer index, DB_REPLICA servers are likely to result in writer index handles. That fine except that $wmgRemoteMasterDbConfig will be used in codfw to point the writer index to the remote servers. Maybe etcd.php should ignore $crossDCLoads if $lbFactoryConf['templateOverridesByCluster'][$mwLoadName]['is static'] is set.

Even the write-active es6 cluster has weight 10 for the writer index, and that cluster does not even have the option of pointing the writer index to a local server. Maybe the weight should be 0, just like es7.

I guess the caching is good enough for the access patterns of HTTP GET requests that this does not result in cross-DC connections that often for now.

aaron mentioned this in T375125: Add $wgExternalStoreConf to support mapping logical clusters to physical clusters.Sep 18 2024, 6:57 PM
aaron added a comment.Sep 19 2024, 6:50 AM

Actually, it looks like wmfApplyEtcdDBConfig() already hard codes 0 load when using remote servers for the writer index. This explains the low log rate. So the only thing left to do here is just disable the primary fallback logic for the 'is static' case in ExternalStoreDB.

gerritbot added a comment.Sep 19 2024, 7:06 AM

Change #1074061 had a related patch set uploaded (by Aaron Schulz; author: Aaron Schulz):

[mediawiki/core@master] externalstore: disable DB_PRIMARY fallback for static data clusters

https://gerrit.wikimedia.org/r/1074061

gerritbot added a project: Patch-For-Review.Sep 19 2024, 7:06 AM
aaron closed this task as Resolved.Oct 1 2024, 4:35 PM

Closing, since the log spam is gone. The remaining patch falls under the "possible general improvements" category.

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits