Once T289612 is deployed (probably in MediaWiki_1.38/wmf.6), we can reindex the relevant wikis, to activate ICU normalization, ICU folding, and homoglyph normalization.
Acceptance Criteria
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Open | None | T219550 [EPIC] Harmonize language analysis across languages | |||
| Resolved | Gehel | T272606 [EPIC] Unpack all Elasticsearch analyzers | |||
| Resolved | TJones | T289612 Unpack Hindi, Irish, Norwegian Elasticsearch Analyzers | |||
| Resolved | TJones | T294257 Reindex Hindi, Irish, Norwegian wikis to enable unpacked versions |
Full write up on Mediawiki.
Summary
@MPhamWMF, move to this to "needs reporting" when you are done looking at the summary/write up.
Feedback or questions from anyone are welcome, of course!
Thanks, @TJones ! This was really interesting. It looks like the Hindi zero results are quite high, which also seems to correlate with hiwiki having a relatively low Search Engagement as well. I wonder which other wikis having a better Latin to native transliterater would help with.
It's definitely an issue that we could try to address. There's an open ticket for Georgian (from Latin/Cyrillic): T127003. And there's a closed ticket with romanization in the title but DWIM ("wrong keyboard") in the description: T245677—I closed it because the diff between wrong keyboard and transliteration was never made clear and it was low priority.
I wouldn't mind getting a proper framework for these kinds of "second-chance" searches and implementing useful modules language-by-language. We have language detection models for wrong-keyboard and wrong-encoding Russian.. though maybe detection isn't the right way to approach it. Suggestions or DWIM-style completion may be a good approach too. I over-engineered an early attempt at wrong-keyboard integration and it flailed and failed.