Page MenuHomePhabricator

Use Unicode Character Folding for accents, punctuation chars in search index
Closed, ResolvedPublic

Description

Author: wikimedia-bugzilla

Description:
It would be desirable for Search and especially for the "Go" functionality
(resolving a page title to an actual page without an intermediate search) to use
all sensible Unicode Foldings on the searched titles.

Unicode Character Foldings define[1] string transformations for making two
strings search-equivalent (different from Unicode normalizations, which make
strings content-equivalent). The folded title should not be stored instead of
the original title but in addition to it, and when searching, the comparisons
should be made between a folded search string and the folded title.

We already do certain forms of folding, such as case insensitivity, but we could
benefit from the full set of foldings, such as eliminating the difference
between minus and dashes and more.

[1] http://www.unicode.org/unicode/reports/tr30/


Version: unspecified
Severity: major

Details

Reference
bz4430

Event Timeline

bzimport raised the priority of this task from to Medium.Nov 21 2014, 9:00 PM
bzimport set Reference to bz4430.
bzimport added a subscriber: Unknown Object (MLST).
brion added a comment.May 19 2008, 5:27 PM
  • Bug 14180 has been marked as a duplicate of this bug. ***
brion added a comment.Dec 28 2008, 9:14 PM

De-assigning, as no activity in 3 years. Still a good idea though! :)

K-form normalization would be easy to apply (since UtfNormal class already implements it); other folding may require more coding.

demon added a comment.Sep 3 2009, 1:53 AM
  • Bug 4379 has been marked as a duplicate of this bug. ***
demon added a comment.Sep 7 2009, 3:42 PM
  • Bug 20529 has been marked as a duplicate of this bug. ***
TJones closed this task as Resolved.Oct 10 2017, 7:36 PM
TJones claimed this task.
TJones added a subscriber: TJones.

I stumbled across this one today, and I'm going to close it because this functionality is generally available and specifically activated on most wikis. For example, searching for Ⱥȴƃǝṟʈ Ɛĭñṥẗȅɨǹ in the Go box brings up the right suggestion (Albert Einstein) from the autocomplete, and hitting return (without selecting the suggestion) rolls over to the right article. Tested on English, French, Russian, Chinese, Arabic, Igbo, Bosnian, Hawaiian, German, and Bambara Wikipedias.

Restricted Application added projects: Discovery, Discovery-Search. · View Herald TranscriptOct 10 2017, 7:36 PM