Page MenuHomePhabricator
Create Task
Maniphest T222819

"missing replica" error messages in production logs
Closed, ResolvedPublic
Actions

Description

While reviewing logs related to cloudelastic turned up these logs which have been low volume but relatively consistent through the last month. These all come from lower volume wikis. Based on the stack trace this most likely has something to do with how OtherIndex writes request a connection for commonswiki's servers instead of their own.

Missing replica <codfw>, have <>

#0 /srv/mediawiki/php-1.34.0-wmf.3/extensions/CirrusSearch/includes/Connection.php(167): CirrusSearch\Assignment\MultiClusterAssignment->getServerList(string)
#1 /srv/mediawiki/php-1.34.0-wmf.3/extensions/Elastica/includes/ElasticaConnection.php(78): CirrusSearch\Connection->getServerList()
#2 /srv/mediawiki/php-1.34.0-wmf.3/extensions/Elastica/includes/ElasticaConnection.php(62): ElasticaConnection->getClient()
#3 /srv/mediawiki/php-1.34.0-wmf.3/extensions/CirrusSearch/includes/Connection.php(133): ElasticaConnection->setConnectTimeout(integer)
#4 /srv/mediawiki/php-1.34.0-wmf.3/extensions/CirrusSearch/includes/Connection.php(113): CirrusSearch\Connection->__construct(CirrusSearch\HashSearchConfig, string)
#5 /srv/mediawiki/php-1.34.0-wmf.3/extensions/CirrusSearch/includes/Connection.php(328): CirrusSearch\Connection::getPool(CirrusSearch\HashSearchConfig, string)
#6 /srv/mediawiki/php-1.34.0-wmf.3/extensions/CirrusSearch/includes/Job/Job.php(231): CirrusSearch\Connection::getClusterConnections(array, CirrusSearch\HashSearchConfig)
#7 /srv/mediawiki/php-1.34.0-wmf.3/extensions/CirrusSearch/includes/Job/ElasticaWrite.php(103): CirrusSearch\Job\Job->decideClusters()
#8 /srv/mediawiki/php-1.34.0-wmf.3/extensions/CirrusSearch/includes/Job/Job.php(100): CirrusSearch\Job\ElasticaWrite->doJob()
#9 /srv/mediawiki/php-1.34.0-wmf.3/extensions/CirrusSearch/includes/OtherIndexes.php(166): CirrusSearch\Job\Job->run()
#10 /srv/mediawiki/php-1.34.0-wmf.3/extensions/CirrusSearch/includes/OtherIndexes.php(145): CirrusSearch\OtherIndexes->runUpdates(Title, array)
#11 /srv/mediawiki/php-1.34.0-wmf.3/extensions/CirrusSearch/includes/Job/OtherIndex.php(70): CirrusSearch\OtherIndexes->updateOtherIndex(array)
#12 /srv/mediawiki/php-1.34.0-wmf.3/extensions/CirrusSearch/includes/Job/Job.php(100): CirrusSearch\Job\OtherIndex->doJob()
#13 /srv/mediawiki/php-1.34.0-wmf.3/extensions/EventBus/includes/JobExecutor.php(66): CirrusSearch\Job\Job->run()
#14 /srv/mediawiki/rpc/RunSingleJob.php(77): JobExecutor->execute(array)
#15 {main}

Potentially related error:

Error in one or more bulk request actions:

update: /commonswiki_file/page/10816880 caused no such index and [action.auto_create_index] contains [-*] which forbids automatic creation of the index [index: commonswiki_file]

Details

SubjectRepoBranchLines +/-
Protect HashSearchConfig from incorrect flagsmediawiki/extensions/CirrusSearchmaster+90 -42
Uniquely identify connections in connection poolmediawiki/extensions/CirrusSearchwmf/1.34.0-wmf.4+84 -8
Uniquely identify connections in connection poolmediawiki/extensions/CirrusSearchmaster+84 -8
Customize query in gerrit

Event Timeline

EBernhardson created this task.May 8 2019, 4:46 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 8 2019, 4:46 PM
EBernhardson claimed this task.May 8 2019, 4:47 PM
EBernhardson added a project: Discovery-Search (Current work).
EBernhardson updated the task description. (Show Details)May 8 2019, 4:59 PM
gerritbot added a comment.May 8 2019, 5:01 PM

Change 508865 had a related patch set uploaded (by EBernhardson; owner: EBernhardson):
[mediawiki/extensions/CirrusSearch@master] Uniquely identify connections in connection pool

https://gerrit.wikimedia.org/r/508865

gerritbot added a project: Patch-For-Review.May 8 2019, 5:01 PM
EBernhardson moved this task from Incoming to Needs review on the Discovery-Search (Current work) board.May 8 2019, 5:35 PM
gerritbot added a comment.May 9 2019, 8:45 AM

Change 508865 merged by jenkins-bot:
[mediawiki/extensions/CirrusSearch@master] Uniquely identify connections in connection pool

https://gerrit.wikimedia.org/r/508865

ReleaseTaggerBot added a project: MW-1.34-notes (1.34.0-wmf.5; 2019-05-14).May 9 2019, 9:00 AM
Aklapper added a project: CirrusSearch.May 9 2019, 11:05 AM
Restricted Application edited projects, added Discovery-Search; removed Discovery-Search (Current work). · View Herald TranscriptMay 9 2019, 11:05 AM
EBernhardson edited projects, added Discovery-Search (Current work); removed Discovery-Search.May 9 2019, 2:30 PM
gerritbot added a comment.May 9 2019, 4:46 PM

Change 509111 had a related patch set uploaded (by EBernhardson; owner: EBernhardson):
[mediawiki/extensions/CirrusSearch@wmf/1.34.0-wmf.4] Uniquely identify connections in connection pool

https://gerrit.wikimedia.org/r/509111

gerritbot added a comment.May 9 2019, 11:32 PM

Change 509111 merged by jenkins-bot:
[mediawiki/extensions/CirrusSearch@wmf/1.34.0-wmf.4] Uniquely identify connections in connection pool

https://gerrit.wikimedia.org/r/509111

EBernhardson added a comment.May 9 2019, 11:54 PM

Fix was deployed at 16:47, saw a few Missing replica <codfw>, have <> log messages immediately after. I'm fairly certain the patch above fixes the no such index error, but this one may still be a problem. Will need to monitor logs before declaring this a win.

ReleaseTaggerBot edited projects, added MW-1.34-notes (1.34.0-wmf.4; 2019-05-07); removed MW-1.34-notes (1.34.0-wmf.5; 2019-05-14).May 10 2019, 12:00 AM
gerritbot added a comment.May 10 2019, 4:06 PM

Change 509454 had a related patch set uploaded (by EBernhardson; owner: EBernhardson):
[mediawiki/extensions/CirrusSearch@master] Protect HashSearchConfig from incorrect flags

https://gerrit.wikimedia.org/r/509454

gerritbot added a comment.May 10 2019, 11:11 PM

Change 509454 merged by jenkins-bot:
[mediawiki/extensions/CirrusSearch@master] Protect HashSearchConfig from incorrect flags

https://gerrit.wikimedia.org/r/509454

ReleaseTaggerBot edited projects, added MW-1.34-notes (1.34.0-wmf.5; 2019-05-14); removed MW-1.34-notes (1.34.0-wmf.4; 2019-05-07).May 11 2019, 12:00 AM
Ladsgroup removed a project: Patch-For-Review.May 21 2019, 10:08 PM
EBernhardson added a comment.May 24 2019, 3:11 PM

No matches in last 7 days, this looks resolved.

EBernhardson moved this task from Needs review to Needs Reporting on the Discovery-Search (Current work) board.May 24 2019, 3:11 PM
debt closed this task as Resolved.May 28 2019, 11:51 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL