See T176197#3737728 and following comments.
After discussions on Wikipedia and Wiktionary, the general consensus is that we should enable the hiragana to katakana (H2K) mapping for French, Russian, Italian, and Swedish because it has a small effect but seems useful.
With the new ICU tokenizer, the H2K mapping causes problems and does very little positive, so we should just turn it off.
| Subject | Repo | Branch | Lines +/- | |
|---|---|---|---|---|
| Disable Hiragana-to-Katakana Mapping | mediawiki/extensions/CirrusSearch | master | +16 -770 |
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | TJones | T176197 Allow hiragana searches to find katakana results and vice versa | |||
| Invalid | None | T218613 {EPIC} Search: Local Impact. Making making bigger improvements for smaller (or underrepresented) communities | |||
| Open | None | T219550 [EPIC] Harmonize language analysis across languages | |||
| Resolved | TJones | T180387 𝖤̶𝗇̶𝖺̶𝖻̶𝗅̶𝖾̶ Disable hiragana/katakana mapping |
Not directly part of this ticket, but part of the related discussion: we should decide whether we should keep going down the list of unpacked analyzers, and whether we should pro-actively unpack the other analyzers, and whether we should just enable it for languages where it has a small but positive impact (and doesn't cause problems with the analyzer).
We'll make an announcement to the various wikis village pumps for this upcoming work (we're not sure of exact timelines) and we'll post again before we release the changes into production.
While we had planned to expand the deployment of the hiragana-to-katakana mapping from English to most other languages (though not Japanese), testing revealed that doing the mapping pre-tokenization interfered with the new ICU tokenizer's ability to parse Japanese text (on non-Japanese wikis).
Enabling the mapping after tokenization didn't do much because there aren't that many cross-kana terms in the dictionary used by the ICU tokenizer. It's also very expensive in the off-the-shelf form (around +10% to load/reindex times), or it would require development and deployment of a new filter—neither of which seems worth it.
In the spirit of cross-language analysis harmonization—and because it's now clear that it's interfering with the ICU tokenizer—I'm removing the cross-kana mapping from the English analysis chain.
Full write up on Mediawiki.
New patch to follow shortly.
Change #1020719 had a related patch set uploaded (by Tjones; author: Tjones):
[mediawiki/extensions/CirrusSearch@master] Disable Hiragana-to-Katakana Mapping
Change #1020719 merged by jenkins-bot:
[mediawiki/extensions/CirrusSearch@master] Disable Hiragana-to-Katakana Mapping