Page MenuHomePhabricator

Fix: "Warning: Search backend error during .. took .." (tracking)
Open, LowestPublic

Description

https://logstash.wikimedia.org/#/dashboard/elasticsearch/mediawiki-error – search for "Search backend error during". As of July 2018, about 11,000 recorded instances of this error per month.

Normalised:

Search backend error during {queryType} search for '{query}' after {tookMs}: {error_message}

Sample (1.32.0.wmf-14)
Search backend error during comp_suggest search for 'audit' after 1: : 

elasticTookMs: "0"
error_message: ":"
hitsOffset: "0"
hitsReturned: "0"
hitsTotal: "-"

level:WARNING
channel:CirrusSearch
url: /w/api.php?action=opensearch&search=audit&limit=15
reqId: W2CfuApAEMIAADlGDK8AAAAK
Sample (1.25wmf23)
Warning: Search backend error during degraded_full_text search
 for '"nicht antretbar*" OR "nicht antreten*" OR "nicht erstattbar*" OR "nicht erstatten*" OR "nicht erstattet*" OR "nicht stornierbar*" OR "nicht stornieren*" OR "nicht storniert*"'
 after 62. 
  Parse error on ' or   OR   OR   OR   OR   OR   or ': Encountered " <OR> "OR "" at line 1, column 11.
[Called from CirrusSearch\ElasticsearchIntermediary::failure in /srv/mediawiki/php-1.25wmf23/extensions/CirrusSearch/includes/ElasticsearchIntermediary.php at line 98] in /srv/mediawiki/php-1.25wmf23/includes/debug/MWDebug.php on line 300

Related Objects

StatusAssignedTask
OpenNone
DuplicateNone
Resolved Manybubbles
ResolvedEBernhardson
ResolvedEBernhardson
DuplicateNone
DuplicateNone
DeclinedNone
Opendcausse
Resolveddcausse
Resolveddcausse
Resolveddcausse
Resolveddcausse
Resolveddcausse
Resolveddcausse
Resolveddcausse
Resolveddcausse
Resolveddcausse
Opendcausse
Resolveddcausse
ResolvedCommunityTechBot
Resolveddcausse
Resolveddcausse
Resolveddcausse
Duplicatedcausse
Resolveddcausse
Resolveddcausse
ResolvedJohan
Opendcausse
Opendcausse

Event Timeline

Krinkle created this task.Apr 2 2015, 9:13 AM
Krinkle raised the priority of this task from to Needs Triage.
Krinkle updated the task description. (Show Details)
Krinkle added a subscriber: Krinkle.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 2 2015, 9:13 AM

3 tasks here:

  1. So the spacing one we're pretty aware of (and have tried to fix already). I could've sworn I had a task for this but I guess not.
  2. I was just about to file an issue for the OR parse error. That'll be its own task.
  3. The highlighting one is bad and its own thing as well.
demon triaged this task as High priority.Apr 3 2015, 5:51 PM

Raising priority too because these are really spammy (and not easily grouped because of the different queries so they don't show up as one item in fatalmonitor)

bd808 edited projects, added Discovery; removed MediaWiki-Core-Team.Apr 7 2015, 4:56 PM
Manybubbles moved this task from Needs triage to Search on the Discovery board.May 7 2015, 7:56 PM

This is caused by @Jdouglas's change to pull out phrase prefixes. This query was never going to work properly anyway - but now it blows up!

I can make it stop blowing up but I can't get it to actually do what the user wants without rewriting the query parser. Which we want to do any way, but we just can't do yet.

hashar added a subscriber: hashar.Jun 1 2015, 2:45 PM

Seems the last task to be completed is T95020 :-}

I'm pretty sure we got this, yeah. Is it still coming up?

I'm also slowly slowly working through a replacement for query_string which should fix this better.

hashar added a comment.Jun 1 2015, 4:13 PM

I have no idea whether it still occurs. I am not sure how to trigger the error nor how to search for it in logstash.wikimedia.org :-/

Manybubbles closed this task as Resolved.Jun 2 2015, 5:02 PM
hashar reopened this task as Open.Jun 2 2015, 7:52 PM

The task detail had the search instruction for logstash: https://logstash.wikimedia.org/#/dashboard/elasticsearch/hhvm search for "ElasticsearchIntermediary"

demon added a comment.Jun 30 2015, 6:44 PM

This was the tracking task for the 3 separate subtasks. Please leave this one open until all 3 are closed. The one with "OR" isn't done.

hashar renamed this task from Fix: "Warning: Search backend error during .. took .." to Fix: "Warning: Search backend error during .. took .." (tracking).Jun 30 2015, 9:38 PM
hashar added a project: Tracking-Neverending.
greg removed Manybubbles as the assignee of this task.Aug 24 2015, 6:56 PM
greg added a subscriber: greg.

Unassigning from Nik, assuming that is correct :)

Deskana moved this task from Needs triage to Search on the Discovery board.Sep 23 2015, 4:41 AM

@EBernhardson does not think we get too many of these in the logs these days, so I'm lowering priority based on that.

Deskana moved this task from Search to Ops on the Discovery board.Nov 24 2015, 6:10 PM
demon added a comment.Nov 24 2015, 6:16 PM

Interesting, I wonder why less instances of it...maybe people have given up?

Deskana lowered the priority of this task from High to Low.Nov 24 2015, 6:19 PM

Lowering priority to reflect the reality of the team's prioritisation.

actually they still exist, they are just shunted off into the CirrusSearch logs, instead of spamming up hhvm.log. Still something to fix, but it wont be annoying other teams looking at general logs by spamming those up.

Deskana moved this task from Uncategorised to Technical on the CirrusSearch board.Dec 31 2015, 5:11 AM
demon removed a subscriber: demon.Feb 7 2017, 5:52 AM
Restricted Application added a project: Discovery-Search. · View Herald TranscriptFeb 7 2017, 5:52 AM
debt lowered the priority of this task from Low to Lowest.Mar 9 2017, 11:16 PM
debt moved this task from needs triage to later on... on the Discovery-Search board.
debt added a subscriber: debt.

Issue still exists in our query parsing...but not a huge deal. Moving to be dealt with later.

Krinkle removed a subscriber: Krinkle.Mar 10 2017, 12:14 AM
hashar removed a subscriber: hashar.Mar 10 2017, 9:04 AM
Krinkle updated the task description. (Show Details)Jul 31 2018, 6:21 PM

As pointed out by Manybubbles this will be completely fixed once we fully get rid of elasticsearch query_string. This work is being tracked in T185108.

mmodell changed the subtype of this task from "Task" to "Production Error".Aug 28 2019, 11:12 PM