@debt No, the results are simply not appearing. The JSON response is still normal, up to the warning I mentioned above, but the results do not render on screen. The symptoms as listed in the task description still persist. See the screenshot in the task description.

Perhaps this is a purely front-end issue?

Note that you don’t need to change your interface to Bengali to see these effects, and the fact that it is the Bengali keyword for “category” doesn’t seem to matter either. You can search for single characters and get the described behavior.

(Be sure to clear the search box between examples—otherwise you the old results, as @Mahir256 noted above.)

For Bengali, Devanagari, and Gurmukhi nukta, the precomposed versions hang, and the decomposed versions get suggestions:

Oddly, the opposite behavior happens for Latin, Cyrillic, and Greek characters—the precomposed versions work and the decomposed versions hang:

However, when there is no precomposed alternative, the decomposed version works fine (depending on your fonts, the mixed script versions may or may not look right):

So, I’m really not sure what’s going on here, but it looks like it is more than just Indic languages that have the problem, and there seems to be an “expected” form which works, and an “unexpected” form that doesn’t—and the (pre|de)composition difference can break in either direction for a given script.

Gotcha, thanks for the verification, @Mahir256 and the investigation, @TJones both should give us the information we need to get it fixed.

Looking at the network trace when searching for ଡ଼, the result is returned (Q31441900) but for some reason is not displayed. The warning is there, but does not prevent the display of the result. I suspect the frontend, but will dig deeper.

In jquery.ui.suggester.js we've got this:

			if ( typeof requestTerm === 'string' && requestTerm !== self._term ) {
				// Skip request since it does not correspond to the current search term.
				return;
			}

So looks like the term we get from the result may not be the original one but normalized one, because this check fails for the search above, thus ensuring no results.

Confirming:

requestTerm.length
2
self._term.length
1
requestTerm == self._term
false
requestTerm.normalize() == self._term.normalize()
true

So I guess the solution is to use normalization when comparing Unicode strings.

Change 387959 had a related patch set uploaded (by Smalyshev; owner: Smalyshev):
[data-values/value-view@master] Normalize search term before matching against result

https://gerrit.wikimedia.org/r/387959

@Smalyshev, thanks for tracking this one down! That was some weird behavior, but things getting normalized and not matching makes sense.

@Snaterlicious, @hoo, @thiemowmde Do you know why the check is there and what it meant to be doing? @tstarling raised the following concern:

The search term is normalized by the server using $wgContLang->normalize(), which potentially includes transformations beyond NFC, especially if the content language is Arabic or Malayalam. So even if you do client-side NFC using the same version of Unicode as the server, there is at least a hypothetical possibility of a hang.

In T170779#3734809, @Smalyshev wrote:

@Snaterlicious, @hoo, @thiemowmde Do you know why the check is there and what it meant to be doing? @tstarling raised the following concern:

The search term is normalized by the server using $wgContLang->normalize(), which potentially includes transformations beyond NFC, especially if the content language is Arabic or Malayalam. So even if you do client-side NFC using the same version of Unicode as the server, there is at least a hypothetical possibility of a hang.

Replied on gerrit.

Change 390365 had a related patch set uploaded (by Tim Starling; owner: Tim Starling):
[mediawiki/extensions/Wikibase@master] To identify superseded requests, use the requested search term instead of the returned search term

https://gerrit.wikimedia.org/r/390365

Change 390365 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] To identify superseded requests, use the requested search term instead of the returned search term

https://gerrit.wikimedia.org/r/390365

Change 387959 abandoned by Smalyshev:
Normalize search term before matching against result

Reason:
looks like better solution has been found

https://gerrit.wikimedia.org/r/387959

Can we consider this done with https://gerrit.wikimedia.org/r/390365?

@thiemowmde I think it should fix it but can't check since it's not deployed yet (not even on test). So I'd like to keep it open until we can verify the problem is indeed gone.

Seems to be working ok on test.wikidata.org, so closing. If it fails on wikidata when the train moves on, please reopen.

@Mahir256 Please note wmf.8 is still not deployed on www.wikidata.org - see https://phabricator.wikimedia.org/T178635.

Wikidata is still on MediaWiki 1.31.0-wmf.7, while the change is included in 1.31.0-wmf.8.

@Sjoerddebruin @Smalyshev @greg What’s up with https://tools.wmflabs.org/versions/ then? Are dewiki and wikidatawiki versions just not reported correctly there?

In T170779#3771313, @Mahir256 wrote:

@greg What’s up with https://tools.wmflabs.org/versions/ then? Are dewiki and wikidatawiki versions just not reported correctly there?

See T178635, tl;dr: database crash halted the train. We'll get wmf.8 everywhere on Monday.

wikidata is special right now with some stuff @Addshore is doing to kill the wikidata build. That should be up-to-date on Monday as well.

@greg Okay, thank you for the clarification. I will close this task after Monday, then. (Probably should fix the URL I linked to, though.)

This problem is present in other Wikis too. I have noticed it in enWS and bnWS, whenever a link containing these characters (like য়) goes to a non-Wiki site, it does not work, but it works fine when the link goes to another Wiki.

@Hrishikes this task is about search and it already has been fixed. Please submit the linking issue as a separate task.