Page MenuHomePhabricator

Expand homoglyph normalization to more scripts (esp. Greek)
Open, HighPublic

Description

User Story: As a searcher or editor, it can be very hard to tell that fοοlprооf contains two Greek ο's and two Cyrillic о's. While Cyrillic/Latin homoglyphs are the most common, there are plenty of Greek/Latin and Cyrillic/Greek homoglyphs, too, and automatically handling them in search makes life easier.

The current homoglyph plugin is really only built to handle the single one-to-one mapping for Cyrillic/Latin. Expanding it to handle multiple mappings and multi-script tokens (Cyrillic/Latin/Greek tokens do exist) would allow us to add Greek and possibly a few other mappings.

Acceptance Criteria:

  • homoglyph_norm can handle Cyrillic/Latin, Greek/Latin, Cyrillic/Greek and Cyrillic/Latin/Greek tokens.
  • Its possible to specify which scripts are enabled (e.g., only Cyrillic and Latin).
  • Execution speed it acceptable. (More than 2x the time to run of the current plugin would need review. 1.5x is a good goal. 1x would be ideal.)
  • Stretch: Add support for additional scripts, like Armenian.

Event Timeline

I wanted to move this out of my ever-languishing 10% project pile.

TJones triaged this task as High priority.Aug 27 2024, 5:49 PM

Font designer here. One thing to consider is that this may depend on the font used. For instance, PT Serif has Cyrillic capital ka and Latin capital K identical. Whereas Times New Roman has the two letters different.
Minion 3 has Greek capital upsilon different from capital Y. Whereas Times New Roman has the two letters the same.

@SVG-image-maker, you are probably even more sensitive to the subtle differences in letter shapes than I am! But I think many people are not. I've seen vandalized English text where every a, e, and o have been replaced with their Cyrillic counterparts and it looks ɡoofy to me because the kerning is so wrong.. but it seems that many people don't really notice (or maybe they just don't recognize the cause and know how to fix it). I also think that when people cut-n-paste from another source that looks fine in the original font they may not notice that it's a little off in the on-wiki font. And there are cases where I'm pretty sure that the person edited together a word from mixed parts because they didn't know what else to do (to type a specific diacritic) or didn't realize the letters are different (like Ukrainian Cyrillic і vs Latin i). Small fonts on phones also make everything harder to notice.

Anyway, all of my mappings are either really obvious in some reasonable font (A/Α/А) or, when they are more generous, they are based on specific examples I found on-wiki. Bulgarian Wiktionary has "гubеr" (Cyrillic in bold: гubеr) for "ruber", which surprised me, but in a small sans serif font that ge does look enough like an r not to notice when reading at speed. (And it's certainly less distinct than the difference between Latin g and ɡ, which I think most people would also gloss over—though that gets rectified elsewhere since they are both actually forms of the same letter.)

For me, the most important thinɡ about the mappings is that we keep the original string, so that if it is correct for some reason then searching for that exact thing will generally generate a good match.