Completion suggester ignores stress marks
Closed, ResolvedPublic
Actions

Description

From the feedback page for the Completion suggester:

"In the Greek language, we use a stress mark (tonos) on vowels to show which syllable is stressed. The normal search of Wikipedia returns articles that differ from the entered string only on stress, which is the expected behaviour. For example, if you enter ανεμος the first search result is άνεμος. However, the CompletionSuggester does not suggest άνεμος or any other word that starts with ά. It does, however, return ανεμοδαρμένος, ανεμούριο, ανεμοβλογιά, ανεμοστρόβιλος and several other words that start with an alpha without a stress mark. Rentzepopoulos (talk) 11:22, 8 March 2016 (UTC)"

I can confirm that the completion suggester beta feature is not taking these stress marks into consideration when presenting possible matches.

Potentially related task: T75605

Details

	Subject	Repo	Branch	Lines +/-
	Enable ICU Folding on greek wikipedia	operations/mediawiki-config	master	+8 -0
	CompletionSuggester: add support for ICU Folding	mediawiki/extensions/CirrusSearch	master	+40 -10

Customize query in gerrit

Related Objects

Mentioned Here: T75605: No normalization for ancient greek accents in searches

Event Timeline

CKoerner_WMF created this task.Mar 10 2016, 4:42 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 10 2016, 4:42 PM

Putting this into Discovery-Search (Current work) for investigation, as the user indicates this is potentially a regression in the completion suggester.

• Deskana moved this task from Needs triage to On Sprint Board on the Discovery-ARCHIVED board.Mar 12 2016, 1:29 AM

Per Nik analysis in T75605: this is a limitation in the ASCII Folding code we use for prefixsearch and completion suggester.

Folding of ά is handled by the greek stemmer and thus supported in fulltext search.
We do not use a stemmer for search as you type (both prefixsearch and the new completion suggester).

We could switch to ICU Folding which seems to handle a wider range or unicode space.

To sum up, this is not a regression, old prefixsearch uses the same asciifolding and does not support such behavior.

This is always difficult to change the analysis chain, I'd propose to fix the completion suggester analysis first and add an option so we could test on greek wikis first.
An option would allow us to revert back to the old behavior if icu_folding introduces unwanted behaviors.
Then when we are confident that icu_folding is better than ascii_folding we could fix the old prefixsearch analysis config.

NOTE: fixing the old prefixsearch analysis chain might be more complex since we share the same asciifolding filter for multiple purposes and icu_folding lacks the option preserve_original added by Nik to asciifolding.

Change 277249 had a related patch set uploaded (by DCausse):
CompletionSuggester: add support for ICU Folding

https://gerrit.wikimedia.org/r/277249

gerritbot added a project: Patch-For-Review.Mar 14 2016, 2:36 PM

dcausse claimed this task.Mar 14 2016, 2:37 PM

dcausse moved this task from Incoming to Needs review on the Discovery-Search (Current work) board.

EBernhardson moved this task from Needs review to Needs Reporting on the Discovery-Search (Current work) board.Mar 14 2016, 10:30 PM

Change 277249 merged by jenkins-bot:
CompletionSuggester: add support for ICU Folding

https://gerrit.wikimedia.org/r/277249

We think this problem should be solved by the above patches, so resolving this task. This should roll out this week, so we'll get quick feedback from users whether it's working or not.

Change 277477 had a related patch set uploaded (by DCausse):
Enable ICU Folding on greek wikipedia

https://gerrit.wikimedia.org/r/277477

ReleaseTaggerBot added a project: MW-1.27-release (WMF-deploy-2016-03-22_(1.27.0-wmf.18)).Mar 15 2016, 8:02 PM

Change 277477 merged by jenkins-bot:
Enable ICU Folding on greek wikipedia