Add a "reverse" suggestion field to workaround the prefix length limitation (typos suggestion)
Closed, ResolvedPublic
Actions

Description

Today we use a suggestion field that is built against the title and redirect data. This field is used by the phrase suggester with a prefix length of 2. In other words if the typo appears in the first two characters all suggestions generated by the phrase suggester would be bad. Reducing the prefix length can hurt performances, another technique is to build a second suggestion field with a "reverse" filter.

Details

	Subject	Repo	Branch	Lines +/-
	Increase phrase suggester recall	mediawiki/extensions/CirrusSearch	master	+134 -14

Customize query in gerrit

Related Objects
Search...

Status	Assigned	Task
Resolved	• Deskana	T125603 EPIC: Review current ElasticSearch configuration, and use relevance lab to run tests to optimise the configuration to improve search result relevance
Resolved	dcausse	T107006 Add a "reverse" suggestion field to workaround the prefix length limitation (typos suggestion)
Resolved	dcausse	T106692 Allow customization of "Did you mean" suggestions.
Resolved	dcausse	T107427 EPIC: Test if a reverse field can help to display more and better suggestions
Resolved	dcausse	T108034 Fix a bug in PhraseSuggester that prevents us to use the reverse field
Resolved	Gehel	T122697 Upgrade ElasticSearch to 1.7.5
Resolved	Gehel	T127074 cirrus browser tests fail on Vagrant (and probably in other places)
Resolved	EBernhardson	T127831 Upgrade ruflin/elastica to 2.3.1

Event Timeline

dcausse created this task.Jul 27 2015, 7:46 AM

dcausse raised the priority of this task from to Needs Triage.

dcausse updated the task description. (Show Details)

dcausse added projects: CirrusSearch, Discovery-Search (Current work).

dcausse subscribed.

Restricted Application added a project: Discovery-ARCHIVED. · View Herald TranscriptJul 27 2015, 7:46 AM

Restricted Application added a subscriber: Aklapper. · View Herald Transcript

dcausse claimed this task.Jul 28 2015, 10:03 AM

dcausse set Security to None.

dcausse moved this task from Incoming to not in use - please delete on the Discovery-Search (Current work) board.Jul 28 2015, 4:25 PM

Change 227478 had a related patch set uploaded (by DCausse):
WIP: Add reverse suggestion field

https://gerrit.wikimedia.org/r/227478

gerritbot added a project: Patch-For-Review.Jul 28 2015, 4:47 PM

dcausse mentioned this in T106692: Allow customization of "Did you mean" suggestions..Jul 28 2015, 4:59 PM

dcausse added a subtask: T106692: Allow customization of "Did you mean" suggestions..Jul 28 2015, 7:09 PM

We need to test if it's worth a try.

dcausse added a subtask: T107427: EPIC: Test if a reverse field can help to display more and better suggestions.Jul 30 2015, 3:49 PM

There's an issue in PhraseSuggester: https://github.com/elastic/elasticsearch/pull/12670

I will update the patch in cirrus to workaround the bug.

dcausse added a subtask: T108034: Fix a bug in PhraseSuggester that prevents us to use the reverse field.Aug 5 2015, 2:51 PM

dcausse changed the status of subtask T107427: EPIC: Test if a reverse field can help to display more and better suggestions from Open to Stalled.Aug 5 2015, 5:19 PM

dcausse moved this task from Needs review to Incoming on the Discovery-Search (Current work) board.Aug 6 2015, 9:45 AM

Looks like there's no workaround for this bug :(
moving to backlog until we have elasticsearch 1.7.2

EBernhardson removed a project: Discovery-Search (Current work).Aug 18 2015, 5:20 PM

• Deskana closed subtask T106692: Allow customization of "Did you mean" suggestions. as Resolved.Sep 12 2015, 2:41 AM

• Deskana closed subtask T108034: Fix a bug in PhraseSuggester that prevents us to use the reverse field as Resolved.Sep 12 2015, 2:44 AM

• Deskana moved this task from Inbox to Scoring and result ordering on the CirrusSearch board.Dec 31 2015, 5:02 AM

Restricted Application added a subscriber: StudiesWorld. · View Herald TranscriptDec 31 2015, 5:02 AM

• Deskana triaged this task as Medium priority.Dec 31 2015, 5:02 AM

• Deskana subscribed.

We will have elastic 1.7.5 soon in production, I think it'd be nice to finish what we started on this task.
The top query for the last week of January 2016 was Terry Wigan which is certainly a typo to Terry Wogan (a radio and TV broadcaster who died this week).
I think that the reverse field would have helped in this case.