Page MenuHomePhabricator

"missing replica" error messages in production logs
Closed, ResolvedPublic

Description

While reviewing logs related to cloudelastic turned up these logs which have been low volume but relatively consistent through the last month. These all come from lower volume wikis. Based on the stack trace this most likely has something to do with how OtherIndex writes request a connection for commonswiki's servers instead of their own.

Missing replica <codfw>, have <>

#0 /srv/mediawiki/php-1.34.0-wmf.3/extensions/CirrusSearch/includes/Connection.php(167): CirrusSearch\Assignment\MultiClusterAssignment->getServerList(string)
#1 /srv/mediawiki/php-1.34.0-wmf.3/extensions/Elastica/includes/ElasticaConnection.php(78): CirrusSearch\Connection->getServerList()
#2 /srv/mediawiki/php-1.34.0-wmf.3/extensions/Elastica/includes/ElasticaConnection.php(62): ElasticaConnection->getClient()
#3 /srv/mediawiki/php-1.34.0-wmf.3/extensions/CirrusSearch/includes/Connection.php(133): ElasticaConnection->setConnectTimeout(integer)
#4 /srv/mediawiki/php-1.34.0-wmf.3/extensions/CirrusSearch/includes/Connection.php(113): CirrusSearch\Connection->__construct(CirrusSearch\HashSearchConfig, string)
#5 /srv/mediawiki/php-1.34.0-wmf.3/extensions/CirrusSearch/includes/Connection.php(328): CirrusSearch\Connection::getPool(CirrusSearch\HashSearchConfig, string)
#6 /srv/mediawiki/php-1.34.0-wmf.3/extensions/CirrusSearch/includes/Job/Job.php(231): CirrusSearch\Connection::getClusterConnections(array, CirrusSearch\HashSearchConfig)
#7 /srv/mediawiki/php-1.34.0-wmf.3/extensions/CirrusSearch/includes/Job/ElasticaWrite.php(103): CirrusSearch\Job\Job->decideClusters()
#8 /srv/mediawiki/php-1.34.0-wmf.3/extensions/CirrusSearch/includes/Job/Job.php(100): CirrusSearch\Job\ElasticaWrite->doJob()
#9 /srv/mediawiki/php-1.34.0-wmf.3/extensions/CirrusSearch/includes/OtherIndexes.php(166): CirrusSearch\Job\Job->run()
#10 /srv/mediawiki/php-1.34.0-wmf.3/extensions/CirrusSearch/includes/OtherIndexes.php(145): CirrusSearch\OtherIndexes->runUpdates(Title, array)
#11 /srv/mediawiki/php-1.34.0-wmf.3/extensions/CirrusSearch/includes/Job/OtherIndex.php(70): CirrusSearch\OtherIndexes->updateOtherIndex(array)
#12 /srv/mediawiki/php-1.34.0-wmf.3/extensions/CirrusSearch/includes/Job/Job.php(100): CirrusSearch\Job\OtherIndex->doJob()
#13 /srv/mediawiki/php-1.34.0-wmf.3/extensions/EventBus/includes/JobExecutor.php(66): CirrusSearch\Job\Job->run()
#14 /srv/mediawiki/rpc/RunSingleJob.php(77): JobExecutor->execute(array)
#15 {main}

Potentially related error:

Error in one or more bulk request actions:

update: /commonswiki_file/page/10816880 caused no such index and [action.auto_create_index] contains [-*] which forbids automatic creation of the index [index: commonswiki_file]

Details

Related Gerrit Patches:
mediawiki/extensions/CirrusSearch : masterProtect HashSearchConfig from incorrect flags
mediawiki/extensions/CirrusSearch : wmf/1.34.0-wmf.4Uniquely identify connections in connection pool
mediawiki/extensions/CirrusSearch : masterUniquely identify connections in connection pool

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 8 2019, 4:46 PM
EBernhardson updated the task description. (Show Details)May 8 2019, 4:59 PM

Change 508865 had a related patch set uploaded (by EBernhardson; owner: EBernhardson):
[mediawiki/extensions/CirrusSearch@master] Uniquely identify connections in connection pool

https://gerrit.wikimedia.org/r/508865

Change 508865 merged by jenkins-bot:
[mediawiki/extensions/CirrusSearch@master] Uniquely identify connections in connection pool

https://gerrit.wikimedia.org/r/508865

Restricted Application edited projects, added Discovery-Search; removed Discovery-Search (Current work). · View Herald TranscriptMay 9 2019, 11:05 AM

Change 509111 had a related patch set uploaded (by EBernhardson; owner: EBernhardson):
[mediawiki/extensions/CirrusSearch@wmf/1.34.0-wmf.4] Uniquely identify connections in connection pool

https://gerrit.wikimedia.org/r/509111

Change 509111 merged by jenkins-bot:
[mediawiki/extensions/CirrusSearch@wmf/1.34.0-wmf.4] Uniquely identify connections in connection pool

https://gerrit.wikimedia.org/r/509111

Fix was deployed at 16:47, saw a few Missing replica <codfw>, have <> log messages immediately after. I'm fairly certain the patch above fixes the no such index error, but this one may still be a problem. Will need to monitor logs before declaring this a win.

Change 509454 had a related patch set uploaded (by EBernhardson; owner: EBernhardson):
[mediawiki/extensions/CirrusSearch@master] Protect HashSearchConfig from incorrect flags

https://gerrit.wikimedia.org/r/509454

Change 509454 merged by jenkins-bot:
[mediawiki/extensions/CirrusSearch@master] Protect HashSearchConfig from incorrect flags

https://gerrit.wikimedia.org/r/509454

No matches in last 7 days, this looks resolved.

debt closed this task as Resolved.May 28 2019, 11:51 PM