Experimenting with MinT I found that it could work for this purpose but right now, it is too slow and unstable to be usable. I don't know if it's a temporary problem or if it's always slow.

In T424641#11873003, @SBisson wrote:

Experimenting with MinT I found that it could work for this purpose but right now, it is too slow and unstable to be usable. I don't know if it's a temporary problem or if it's always slow.

An alternative could be using a transliteration library, which can work for people and company names that are pronounced the same across languages despite being written in different scripts. Since language sounds don't have a 1-to-1 mapping, we may need to support some "fuzzy" searching to get results.
For example, I did a small test using the demo site of this library: "महात्मा गांधी" was transliterated into "mhaatmaa gaaNdhii", which on English Wikipedia search returned "Mahatma Gandhi" as top result along other relevant ones.

Obviously, this approach is not goig to get a 100% success rate, but considering that it may help in some cases, and the consequences of showing an irrelevant item in the search results is not problematic. It may be a beneficial fallback before just showing "no results".

This issue was reported by a Bangla Wikipedi editor:

1. Go to bn:Special:NewArticle and write ওমর
2. It suggests a few articles that already exist on Bangla Wikipedia. Not a problem, but I was looking for articles that don't already exist here.
3. Now write ওমর আর্তান or ওমর আব্দুলকাদির আর্তান
4. It shows "No subjects found for ওমর আর্তান". However, I was hoping that this article would show up. (The same problem also exists on Special:ContentTranslation.)

This need exploration/decision on what approach to pursue. Could we just ask users to try searching in English as well?

In T424641#12079128, @Nikerabbit wrote:

This need exploration/decision on what approach to pursue. Could we just ask users to try searching in English as well?

That does not seem ideal. I think editors in a given community should be able to participate in their language. Having said that, we can explore how the messaging can hint in that direction. Something along the lines of "No subjects found for X, you can try searching for it in other languages you know that may have better coverage of topics." (with the last part maybe showing only when the query is using non-latin script).

In any case, I think it is better when the system can fix things directly rather than telling users to do it. In this case, I think the options also represent a general improvement to the language offerings for Wikimedia in general, being it a faster MinT that can be used in a similar context or a transliteration service.

I'd recommend check-in with @TJones per recent blog post on this subject: https://diff.wikimedia.org/2026/06/11/do-what-i-mean-second-try-suggestions-for-wrong-keyboard-and-transliterated-search/

Thanks for pinging me, @Nikerabbit! Whew.. there's a lot going on here. I'm not as familiar with Wikidata search, which has some differences from search on Wikipedias and other projects, and I don't recognize the UI from the screenshot. I'm not an expert on Hindi, either. So I guess I'm barely qualified to be talking about this, but I'll give it a shot. (And apparently I have a lot to say and I've almost written a first draft for another blog post!—but that is par for the course.)

It's not clear to me if this is about as-you-type suggestions in the search box, fulltext search results, or some other UI (like the Bangla new article example above), but the issues are similar, though weighted differently in different contexts.

The biggest issue for as-you-type suggestions is latency. While MinT or an LLM or a task-specific transliteration machine-learning model could probably do a good job, we want to round-trip from the browser to the search servers and back for every or almost every letter they type. There's a huge memory-intensive data structure that basically precomputes all possible suggestions to get it done fast enough. We sometimes give it multiple different inputs and merge the outputs, but the core of it is (and needs to be) very fast.

Fulltext search also needs to worry about latency, but not at the "every keystroke" level. Sending off requests to an external LLM, for example, could be too slow.

There can be other compute limitations that are not obvious, like some approaches requiring a separate index (and increasing the number of servers we'd need) or greatly increasing the size of that in-memory data structure for as-you-type suggestions (which would require much beefier servers or many more servers to support). These are specific to particular approaches, and we (the Search Team) sometimes end up trying to come up with ideas that may be a bit less effective while being much less expensive—like the lightweight Latin-to-Devanagari transliteration for as-you-type suggestions from my blog post.

A big concern in the context of transliteration is accuracy, especially when English is involved (either into or out of) because English spelling is so ridiculous. The Latin alphabet also isn't big enough for a lot of languages (including English!), and there is a lack of standards for particular sounds across languages. For example, श (more or less /ʃ/) could be transliterated "sh" in English, "ch" in French, "sch" in German, "x" in Catalan, "š" in Czech, "ś" in Polish, "sj" in Dutch, "sc" in Italian, "ș" in Romanian, "ş" in Turkish, "s" in Hungarian, etc. This can be a big problem for names because names (like "Schumer", Germanic spelling) or the "standard" Latin transliteration of a name (like "Tchaikovsky", French spelling) can come from or through any of various different Latin-using languages.

Pau's transliterated "mhaatmaa gaaNdhii" example above is interesting. For Hindi Devanagari-to-Latin transliteration, I'd post-process that and deduplicate doubled vowels because while it is less accurate as a transliteration, it seems to more common in transliteration to English.

For as-you-type suggestions, we have a smaller search target (titles), so we can correct for one or two typos or transliteration errors (depending on query length) on Wikipedias and other non-Wikidata projects. That doesn't seem to be the case on Wikidata (more on that in a second), but I don't recall the details offhand. For fulltext search, we can't match typos, though we can make "Did you mean?" suggestions that correct some typos. Those aren't always great, and we don't currently check that the offered suggestions actually get any results, which can lead to ridiculous chains where a query gets no results, and we auto-search for a suggestion that gets no results, which may even have a third suggestion that also gets no results. (That's rare, but I have found an infinite loop between two suggestions. Checking for results before making a suggestion is on the longterm list of things to fix, but checking for non-zero results is not free...)

Transliterations are more likely to be not quite right, which is more acceptable for as-you-type suggestions, because if they are terrible, there won't be any additional suggestions, and the searcher will never know. Right now, for fulltext did-you-mean suggestions, I wouldn't want to show any more semi-ridiculous suggestions (like "Did you mean mhaatmaa gaaNdhii", which gets neither fulltext results nor as-you-type suggestions for me).

The specific example at hand is also interesting. चक शूमर seems to be the Hindi transliteration of "Chuck Schumer". Interestingly, his Wikidata item does have Marathi चक शुमर. As a naive first approximation, I'd transliterate (thinking from Hindi-ish to English-ish) चक शूमर as "chuk shuumar" and चक शुमर as "chuk shumar". I might post process final "k" after a vowel to "ck" and deduplicate "uu" to "u", but that's going into English specifically, not necessarily into the Latin alphabet generally. For both examples that gives "chuck shumar", which doesn't still get as-you-type or did-you-mean suggestions for "Chuck Sc'''h'''um'''e'''r" on Wikidata, though it does get both on English Wikipedia.

I'd have to dig into the reasons, (or ask @dcausse), but Hindi चक शूमर and Marathi चक शुमर only differ by one character, and "chuck shumar" is only two characters away from "Chuck Schumer", but we don't get as-you-type suggestions for them on Wikidata, which is disappointing—but I'm sure there's a reason that I've forgotten.

Another complication with Wikidata is that it is so multilingual. I expect many topics relevant to India and Hindi (like साड़ी / sari) to already have Hindi labels, while globally notable but not India- or Hindi-specific topics are more likely not to. (And we can extrapolate that pattern to all languages and geographies.)

We definitely cannot afford to run all possible transliterations (or other mappings, like wrong-keyboard mappings.. see the blog post Niklas mentioned) on all queries. We could check for specific characters to trigger mappings. For example, Devanagari gets Hindi-to-English transliteration mappings and Cyrillic gets Russian-to-QWERTY wrong-keyboard mappings, though that could be annoying to Ukrainian- and Marathi-speaking searchers, for example, and it could be expensive to detect all relevant characters if we eventually developed a lot of them.

We could additionally condition the mappings on interface language or a user preference, which would be more precise, but it would miss opportunities for matches for non–logged-in searchers or searchers who are okay with a default English interface but don't actually primarily search in English.

Another way of looking at the problem is to say that Hindi Wikidata labels are not (yet) as well-developed as English Wikidata labels, and this is a job for the WikiGnomes to eventually improve. That isn't very satisfying by itself, but it does kind of point to another approach. Adding labels doesn't have the latency concerns that searching does, so heavier, slower, more accurate approaches could work to generate labels, though they would probably still want or need a human in the loop, depending on the source of transliterations/labels. An off-the-cuff prioritization heuristic would be items with the most wiki-project links that don't have a label in the target language. (Another could be having a Wikipedia link in the target language, but no label in the language... though just copying the article title to the label could go wrong in some cases.) Anyway, using MinT, or a more robust transliteration library, or an LLM to suggest labels (and possibly validating them with the number of web search engine hits from your favorite search engine or other external validation source) seems reasonable—and I'm sure I'm not the first to think of it.

So, in summary, some ideas for addressing the issue:

(1) WikiGnomes, possibly with automated assistance for prioritization or labelling, fill in missing labels, eventually catching up a target language's labels for globally notable items.

(2) Integrate a heavyweight transliteration/translation tool (MinT, a machine learning– or dictionary-based transliteration library, etc.) into a UI/workflow that can afford the extra latency and appreciates the extra accuracy.

(3) Create or find/port a lightweight transliteration library that has acceptably low latency and acceptably high accuracy and use it to provide alternative inputs into any of our suggestion workflows.

(4) Condition using any tool or library providing transliteration alternatives based on relevance, either through language preferences or character sets used.

(5) Figure out again why Wikidata doesn't do as-you-type typo handling—I'm guessing it's just the default configuration, but I don't recall why—and look into loosening it up for transliteration-type suggestions. (As in, we know this a semi-desperate last-ditch effort, so let's loosen up the matching criteria.)

I'd be interested in working on (3), which would probably also benefit from some work to address (4) and (5)—though I can't commit to a timeline without talking to my team about priorities and such.

(I'll be on vacation until next Wednesday, so if there are any replies or questions that need my attention I'll get back to them then.)

Thanks for the input @TJones. The analysis and possible directions make sense.
Just to provide some context, for this particular case the focus is on providing a fallback set of options as the user types. This means that:

• Users don't get exposed to the transformed (transliterated or translated) query. Thus, it is ok if the result is an approximation that produces relevant results when searching for Wikidata items behind the scenes.
• The final Wikidata items selected will be exposed to the user, but showing less relevant suggestions in this context is not a big issue. So we have room for approaches that are not always 100% right.
• Results are produced as the user types, but for the cross-script approach we may have some extra room since it may only apply when there are no exact matches. We could explore ideas to help the user consider the wait more reasonable. (e.g., communicate to the user that no exact match was found and the system is searching for the term in other languages).

One challenge for the WikiGnomes approach is that the focus here is on article creation. So the items users search are article titles that do not exist yet in their wiki. This limits the audience for the task to multilingual users, and represents a broad space (maybe we could prioritize with some heuristic for relevance that tries to approximate the likelihood for a given topic to be created on a specific wiki).

Thanks @TJones for the detailed breakdown of options and @Pginer-WMF for the transliteration test.

Proposed approach

The most promising near-term path seems to be option 3: a lightweight transliteration library potentially combined with fuzzy matching on Wikidata (option 5).

The two would work together: transliteration converts the non-Latin query to an approximate Latin string, and fuzzy matching bridges the gap between that approximation and the actual Wikidata label.

Constraints by language

• Bangla: this approach should work well. Bangla script fully encodes vowel information, making transliteration reliable enough for fuzzy matching to cover the remaining distance.
• Arabic: short vowels are not written in standard text. For example, فيروز ("Fairuz") would transliterate to something like "fyrwz," with the vowels absent entirely. Fuzzy matching is unlikely to bridge that gap reliably, and a better solution would bring us closer to machine translation.

A complementary approach: MT pre-translation of Wikidata labels offline

Items that have English labels but are missing Arabic or Bengali labels could be pre-translated via MT and stored locally for article guidance topic matching purposes (not added to Wikidata). Real-time search could then do matching against that local multilingual label set, avoiding both the latency of live MT and the vowel problem in transliteration.

Proposed next steps

1. Look into both options (transliteration library + fuzzy matching) or (pre-translation with MT) and determine which one (or combination) is the easiest to implement for Bangla and/or Arabic
   • We don't know if fuzzy matching is a possibility and how much work it would entail for us to do that
   • Transliteration library seems to be "ready to use"?
   • pre-translation of labels seems to be something that we could do following a similar approach as for SX in the past
2. Instrument this part of the flow so we can monitor it
3. Launch the experiment in Bangla and/or Arabic and monitor which % of cases are being solved by the implemented solution

Update:

After discussing the options as a team, here is where we landed:

• Fuzzy matching (option 5): worth exploring as a configuration change on the Wikidata side. @dcausse, is this something that could be adjusted on your side without significant engineering effort? If so, it could serve several purposes beyond this task and would be worth pursuing in parallel.

• Pre-translated Wikidata label dataset: higher effort than initially anticipated, setting aside for now.
• MT via MinT (real-time): the most viable option given implementation speed and range of impact. This is what we are scoping for this task.

Scope

Two parallel tasks:

1. Connect to MinT directly to translate queries in real time and retrieve Wikidata matches for languages where labels are missing. (Updating main task description in this task)

2. Add a message in the search step that informs the user that a search is in progress while translation and result retrieval happens in the background, reducing the likelihood of drop-off during the wait. (created a new task for this: T431532)

I'd have to dig into the reasons, (or ask @dcausse), but Hindi चक शूमर and Marathi चक शुमर only differ by one character, and "chuck shumar" is only two characters away from "Chuck Schumer", but we don't get as-you-type suggestions for them on Wikidata, which is disappointing—but I'm sure there's a reason that I've forgotten.

Fuzzy matching on type-ahead is only handled by the completion suggester which is not available on wikidata. Main reason is that wikidata is an order of magnitude larger than other wikis, it has to suggest labels not page titles and is multilingual. It's unclear if we have the capacity to provide the same data-structure for wikidata but even if we have it it'll be a big chunk of work to make the completion suggester compatible with wikidata.

Fuzzy matching (option 5): worth exploring as a configuration change on the Wikidata side. @dcausse, is this something that could be adjusted on your side without significant engineering effort? If so, it could serve several purposes beyond this task and would be worth pursuing in parallel.

It highly depends on your needs, I understand that you have a search-as-you-type UX and thus I suppose that you would want to use the wbsearchentities API module but with a parameter that would enable fuzzy matching?
It'll certainly have a big impact on latencies that may or may not be acceptable (for instance the search backend time to search for chuck shumar is around 400ms if we don't allow the variation to appear in the first char, it's likely to have p95s around 1s).
But we could certainly explore something, the way to wire this up might be to use the profile param to instruct the backend that you want some fuzzyness.

Then we have to discuss where to apply the fuzzyness, the wikidata completion search is quite complex and hits many different search fields:

• prefix (to match the beginning of the label)
• nearmatch (to match the label as whole)
• all the above with some accent/icu folding variations
• all the above mixed with the target language and its fallbacks

It's around 900 search fields that we have to filter/combine.

Unless you have very precise needs it might take a few iterations to get this correctly because fuzzyness is likely to increase recall greatly adding a lot of noise to the output, it might help to have a dozen examples to help clarify all this.

Few caveats that we may not be able to solve quickly:

• fuzzyness won't be applied to do partial matching: on incomplete user-input, the fuzzy match might always be attempted to the full label string, not all labels starting with it
• we don't have proper normalization for fuzzy matching yet for wikidata, in other words the user input must already be normalized (lower case/accent folding). If the target language is always english and that the transliteration mostly produce lower-case ascii it might not be a problem.

MinT transliteration test: Bengali proper names

Testing with Claude whether MinT can transliterate Bengali proper names into English well enough to find the correct person via a Wikipedia search. 100 names tested across singers, actors, cricketers, politicians, writers, and film stars.

Method: Each Bengali name was searched in Bengali Wikipedia to check if a direct match exists. The MinT transliteration was then searched in English Wikipedia. If the MinT output returns the correct person's Wikipedia article, we treat the transliteration as usable. Wikipedia article titles are used as a proxy for the widely accepted English spelling of each name.

Results

Summary

• ✅ Both Bangla and MinT find the correct person: 71
• ✅ Bangla direct match | ⚠️ MinT needs fuzzy search: 21
• ✅ MinT match only (no Bengali Wikipedia article): 1
• ❌ MinT literal translation failure: 3
• ❌ No English Wikipedia article (cannot assess): 4

Change #1311518 had a related patch set uploaded (by Sbisson; author: Sbisson):

[mediawiki/extensions/ArticleGuidance@master] newarticle: Add MinT translation fallback for topic matching

https://gerrit.wikimedia.org/r/1311518