Page MenuHomePhabricator

Undefined index: _score and _id in CirrusSearch/includes/Query/CompSuggestQueryBuilder.php
Closed, ResolvedPublic

Description

Seen in fatalmonitor

     80 Undefined index: _score in /srv/mediawiki/php-1.31.0-wmf.7/extensions/CirrusSearch/in
cludes/Query/CompSuggestQueryBuilder.php on line 224
     80 Undefined index: _id in /srv/mediawiki/php-1.31.0-wmf.7/extensions/CirrusSearch/inclu
des/Query/CompSuggestQueryBuilder.php on line 223

Details

Related Gerrit Patches:

Event Timeline

dcausse created this task.Nov 17 2017, 4:28 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
dcausse triaged this task as High priority.Nov 17 2017, 4:31 PM

moving to high because this code should not reach group2

dcausse closed this task as Declined.Nov 17 2017, 5:20 PM

It was just a brief spike of 300 errors today around 15:00 UTC, looking at the code I see no obvious reasons why it could happen, except a broken response from elastic.
I'll assume that this error was due to the rolling restart and won't try to hide it by calling isset on the result sets.

Krinkle renamed this task from Undefined index: _score in /srv/mediawiki/php-1.31.0-wmf.7/extensions/CirrusSearch/in cludes/Query/CompSuggestQueryBuilder.php on line 224 to Undefined index: _score and _id in CirrusSearch/includes/Query/CompSuggestQueryBuilder.php.Jul 18 2018, 10:21 PM
Krinkle reopened this task as Open.

Still seen on mediawiki-errors in Logstash: The second and third most frequent PHP error from MediaWiki, in the last 7 days.

2[{exception_id}] {exception_url} ErrorException from line 226 of /srv/mediawiki/php-1.32.0-wmf.12/extensions/CirrusSearch/includes/Query/CompSuggestQueryBuilder.php: PHP Notice: Undefined index: _id8,860 hits
3[{exception_id}] {exception_url} ErrorException from line 227 of /srv/mediawiki/php-1.32.0-wmf.12/extensions/CirrusSearch/includes/Query/CompSuggestQueryBuilder.php: PHP Notice: Undefined index: _score8,280 hits

Change 447567 had a related patch set uploaded (by EBernhardson; owner: EBernhardson):
[operations/puppet@production] Split per-cluster config out of elasticsearch::curator

https://gerrit.wikimedia.org/r/447567

Change 447567 merged by Gehel:
[operations/puppet@production] Split per-cluster config out of elasticsearch::curator

https://gerrit.wikimedia.org/r/447567

dcausse added a comment.EditedJul 30 2018, 1:21 PM

Found a possible reason for this to happen and filed a bug upstream: https://github.com/elastic/elasticsearch/issues/32467
In short the completion suggester code does not handle gracefully shard failures during the fetch phase causing the response received by cirrus to not contain the necessary information.

I'll add a workaround in Cirrus with a note to this bug upstream.

Change 449200 had a related patch set uploaded (by DCausse; owner: DCausse):
[mediawiki/extensions/CirrusSearch@master] Workaround bad response returned by the completion suggester

https://gerrit.wikimedia.org/r/449200

Change 449200 merged by jenkins-bot:
[mediawiki/extensions/CirrusSearch@master] Workaround bad response returned by the completion suggester

https://gerrit.wikimedia.org/r/449200

debt closed this task as Resolved.Jul 31 2018, 5:56 PM
mmodell changed the subtype of this task from "Task" to "Production Error".Aug 28 2019, 11:09 PM