Fix: "Warning: Search backend error during .. took .." (tracking)
Open, LowestPublicPRODUCTION ERROR
Actions

Assigned To

None

Authored By

	Krinkle
	Apr 2 2015, 9:13 AM

Description

https://logstash.wikimedia.org/#/dashboard/elasticsearch/mediawiki-error – search for "Search backend error during". As of July 2018, about 11,000 recorded instances of this error per month. For the most part these errors mean a user submitted an invalid search string.

Normalised:

Search backend error during {queryType} search for '{query}' after {tookMs}: {error_message}

Sample (1.32.0.wmf-14)

Search backend error during comp_suggest search for 'audit' after 1: : 

elasticTookMs: "0"
error_message: ":"
hitsOffset: "0"
hitsReturned: "0"
hitsTotal: "-"

level:WARNING
channel:CirrusSearch
url: /w/api.php?action=opensearch&search=audit&limit=15
reqId: W2CfuApAEMIAADlGDK8AAAAK

Sample (1.25wmf23)

Warning: Search backend error during degraded_full_text search
 for '"nicht antretbar*" OR "nicht antreten*" OR "nicht erstattbar*" OR "nicht erstatten*" OR "nicht erstattet*" OR "nicht stornierbar*" OR "nicht stornieren*" OR "nicht storniert*"'
 after 62. 
  Parse error on ' or   OR   OR   OR   OR   OR   or ': Encountered " <OR> "OR "" at line 1, column 11.
[Called from CirrusSearch\ElasticsearchIntermediary::failure in /srv/mediawiki/php-1.25wmf23/extensions/CirrusSearch/includes/ElasticsearchIntermediary.php at line 98] in /srv/mediawiki/php-1.25wmf23/includes/debug/MWDebug.php on line 300

Related Objects
Search...

Status	Subtype	Assigned	Task
Open	PRODUCTION ERROR	None	T94814 Fix: "Warning: Search backend error during .. took .." (tracking)
Duplicate	PRODUCTION ERROR	None	T95020 Parse errors with ORs in query
Resolved	PRODUCTION ERROR	• Manybubbles	T95021 Index out of bounds trying to highlight title/text
Resolved	PRODUCTION ERROR	EBernhardson	T75374 Parse errors with extra (ideographic) spaces in query
Resolved		EBernhardson	T106336 Long unicode queries cause hhvm warning: "determinizing automaton would result in more than 10000 states"
Duplicate	PRODUCTION ERROR	None	T90775 support OR between intitle: operators
Duplicate		None	T98873 [epic] CirrusSearch: Abandon query_string
Declined		None	T99261 Implement query_string_plus_plus_plus query parsing
Open		None	T185108 Refactor query parsing logic
Resolved		dcausse	T186879 Unify SearchKeywords implementation
Resolved		dcausse	T188530 Externalize the parsing logic from SimpleKeywordFeature and FullTextQueryStringQueryBuilder
Resolved		dcausse	T189880 Introduce SearchQuery
Resolved		dcausse	T197774 Add a way to classify queries
Resolved		dcausse	T198860 Deprecate SearchEngine::replacePrefixes
Resolved		dcausse	T204255 Introduce rewrite/second-try strategies
Resolved		dcausse	T190130 HighlightBuilder: Introduce a way to control the highlighting behaviors
Resolved		dcausse	T228626 Extract interfaces and base classes from SearchResultSet and SearchResult
Resolved		dcausse	T190131 Externalize the phrase suggest builder (DYM)
Open		None	T190132 Introduce keyword interfaces dedicated per usage
Resolved		dcausse	T193392 Deprecate global namespace handling of the prefix keyword (reverted)
Resolved		CommunityTechBot	T194475 Searching by prefix no longer works
Resolved		dcausse	T195305 Add support for boosting keywords
Resolved		dcausse	T195788 Add support for Filtering keyword
Resolved		dcausse	T195815 The prefix keyword should augment and not override the list of requested namespaces
Duplicate		dcausse	T195881 Add a way for keywords to control highlighting
Resolved		dcausse	T191876 Cirrus profile management should provide a standard way to override profile content
Resolved		dcausse	T198318 Change how SpecialSearch/SearchEngine handles the 'prefix' URI param set by the InputBox extension
Resolved		Johan	T198851 CL support to validate a small change in the Special:Search user interface
Open		None	T164589 incategory search does not support OR (union) operator

Event Timeline

Krinkle created this task.Apr 2 2015, 9:13 AM

Krinkle raised the priority of this task from to Needs Triage.

Krinkle updated the task description. (Show Details)

Krinkle added projects: CirrusSearch, Wikimedia-production-error.

Krinkle subscribed.

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 2 2015, 9:13 AM

3 tasks here:

So the spacing one we're pretty aware of (and have tried to fix already). I could've sworn I had a task for this but I guess not.
I was just about to file an issue for the OR parse error. That'll be its own task.
The highlighting one is bad and its own thing as well.

Raising priority too because these are really spammy (and not easily grouped because of the different queries so they don't show up as one item in fatalmonitor)

• demon added a subtask: T75374: Parse errors with extra (ideographic) spaces in query.Apr 3 2015, 5:57 PM

• Jdouglas added a project: MediaWiki-Core-Team.Apr 3 2015, 5:57 PM

• Jdouglas set Security to None.

bd808 edited projects, added Discovery-ARCHIVED; removed MediaWiki-Core-Team.Apr 7 2015, 4:56 PM

• Manybubbles closed subtask T95021: Index out of bounds trying to highlight title/text as Resolved.Apr 27 2015, 8:35 PM

• Manybubbles moved this task from Needs triage to Search on the Discovery-ARCHIVED board.May 7 2015, 7:56 PM

• Manybubbles added a project: Discovery-Search (Current work).May 8 2015, 7:44 PM

• Manybubbles claimed this task.May 8 2015, 7:48 PM

• Manybubbles moved this task from Incoming to not in use - please delete on the Discovery-Search (Current work) board.

• Manybubbles moved this task from Search to On Sprint Board on the Discovery-ARCHIVED board.May 11 2015, 5:21 PM

This is caused by @Jdouglas's change to pull out phrase prefixes. This query was never going to work properly anyway - but now it blows up!

I can make it stop blowing up but I can't get it to actually do what the user wants without rewriting the query parser. Which we want to do any way, but we just can't do yet.

• Manybubbles moved this task from not in use - please delete to Needs review on the Discovery-Search (Current work) board.May 12 2015, 7:28 PM

• Manybubbles mentioned this in rECIRb3a43cc293bb: Fix errors that look like OR OR OR OR.May 12 2015, 8:23 PM

• Manybubbles mentioned this in rMEXTeffe6eab8a71: Updated mediawiki/extensions Project: mediawiki/extensions/CirrusSearch….

• demon mentioned this in rECIRf2c0f2465c0a: Fix errors that look like OR OR OR OR.May 12 2015, 8:26 PM

• Manybubbles moved this task from Needs review to Needs Reporting on the Discovery-Search (Current work) board.May 12 2015, 8:58 PM

EBernhardson closed subtask T75374: Parse errors with extra (ideographic) spaces in query as Resolved.May 12 2015, 9:33 PM

Seems the last task to be completed is T95020 :-}

hashar moved this task from Untriaged to Jan 2019 / 1.33.wmf13–14 on the Wikimedia-production-error board.Jun 1 2015, 2:45 PM

I'm pretty sure we got this, yeah. Is it still coming up?

I'm also slowly slowly working through a replacement for query_string which should fix this better.

I have no idea whether it still occurs. I am not sure how to trigger the error nor how to search for it in logstash.wikimedia.org :-/

• Manybubbles closed this task as Resolved.Jun 2 2015, 5:02 PM

The task detail had the search instruction for logstash: https://logstash.wikimedia.org/#/dashboard/elasticsearch/hhvm search for "ElasticsearchIntermediary"

hashar merged a task: T95020: Parse errors with ORs in query.Jun 2 2015, 7:52 PM

This was the tracking task for the 3 separate subtasks. Please leave this one open until all 3 are closed. The one with "OR" isn't done.

• demon reopened subtask T95020: Parse errors with ORs in query as Open.Jun 30 2015, 6:45 PM

hashar renamed this task from Fix: "Warning: Search backend error during .. took .." to Fix: "Warning: Search backend error during .. took .." (tracking).Jun 30 2015, 9:38 PM

hashar added a project: Tracking-Neverending.

Unassigning from Nik, assuming that is correct :)

• demon moved this task from Jan 2019 / 1.33.wmf13–14 to Dec2019/1.35.wmf.10+ on the Wikimedia-production-error board.Sep 2 2015, 5:17 PM

• ksmith added a project: Essential-Work.Sep 11 2015, 10:26 PM

• Deskana closed subtask T106336: Long unicode queries cause hhvm warning: "determinizing automaton would result in more than 10000 states" as Resolved.Sep 11 2015, 10:29 PM

• Deskana removed a project: Discovery-Search (Current work).Sep 12 2015, 2:58 AM

• ksmith moved this task from On Sprint Board to Needs triage on the Discovery-ARCHIVED board.Sep 22 2015, 8:27 PM

• Deskana moved this task from Needs triage to Search on the Discovery-ARCHIVED board.Sep 23 2015, 4:41 AM

@EBernhardson does not think we get too many of these in the logs these days, so I'm lowering priority based on that.

• Deskana moved this task from Search to Ops on the Discovery-ARCHIVED board.Nov 24 2015, 6:10 PM

Interesting, I wonder why less instances of it...maybe people have given up?

Lowering priority to reflect the reality of the team's prioritisation.

• demon added a subtask: T90775: support OR between intitle: operators.Nov 24 2015, 6:24 PM

actually they still exist, they are just shunted off into the CirrusSearch logs, instead of spamming up hhvm.log. Still something to fix, but it wont be annoying other teams looking at general logs by spamming those up.

• Deskana moved this task from Inbox to Technical on the CirrusSearch board.Dec 31 2015, 5:11 AM

• demon unsubscribed.Feb 7 2017, 5:52 AM

Restricted Application added a project: Discovery-Search. · View Herald TranscriptFeb 7 2017, 5:52 AM

Issue still exists in our query parsing...but not a huge deal. Moving to be dealt with later.

Krinkle unsubscribed.Mar 10 2017, 12:14 AM

hashar unsubscribed.Mar 10 2017, 9:04 AM

Liuxinyu970226 subscribed.Mar 31 2017, 2:43 PM

Krinkle updated the task description. (Show Details)Jul 31 2018, 6:21 PM

As pointed out by Manybubbles this will be completely fixed once we fully get rid of elasticsearch query_string. This work is being tracked in T185108.

dcausse added a subtask: T185108: Refactor query parsing logic.Jul 31 2018, 6:54 PM

Krinkle moved this task from Dec2019/1.35.wmf.10+ to Apr 2019 / 1.33.wmf.25+ on the Wikimedia-production-error board.Sep 17 2018, 5:30 PM

Krinkle moved this task from Apr 2019 / 1.33.wmf.25+ to Older on the Wikimedia-production-error board.Sep 19 2018, 12:59 AM

• mmodell changed the subtype of this task from "Task" to "Production Error".Aug 28 2019, 11:12 PM

EBernhardson updated the task description. (Show Details)Sep 24 2020, 7:08 PM

Untagging as prod error since this is not a fatal error, runtime PHP warning, HTTP timeout, or HTTP server error.

It no longer appears on the errors dashboard, but can still be found on the "mediawiki" dashboard and/or on search-specific dashboards. It continues to be important and has impact on people searching, but seems well-contained to not be an operational risk and/or has fallbacks in place.

Closing out low/est priority tasks over 6 months old with no activity within last 6 months in order to clean out the backlog of tickets we will not be addressing in the near term. Please feel free to reopen if you think a ticket is important, but bare in mind that given current priorities and resourcing, it is unlikely for the Search team to pick up these tasks for the indefinite future. We hope that the requested changes have either been addressed by or made irrelevant by work the team has done or is doing -- e.g. upgrading Elasticsearch to a newer version will solve various ES-related problems -- or will be subsumed by future work in a more generalized way.

Re-opening tasks and removing from team workboard per IRC feedback given yesterday and discussion with MPham.

Fix: "Warning: Search backend error during .. took .." (tracking)Open, LowestPublicPRODUCTION ERRORActions

Description

Related ObjectsSearch...

Event Timeline

Fix: "Warning: Search backend error during .. took .." (tracking)
Open, LowestPublicPRODUCTION ERROR
Actions

Related Objects
Search...