Removing MediaWiki-Search as that's not used on Wikimedia wikis (but CirrusSearch).

Interesting results indeed when using a backtick:

I just want to clarify the issue. When our contributors write an article first in MS Word (which happens often, because they feel more comfortable to write in Word than in Visual Editor) the apostroph on the keyboard (', u+0027) will be turned automatically by Word into another apostroph (ʼ, u+02BC).

Unfortunately this happens also when they write the page title.

The result: if for example a user search for keyword ndru'u (with u+0027) it will not find all words ndruʼu (with u+02BC) and visa versa. This happens across Wikipedia, Wiktionary and Wikibooks.

I ran a robot some times ago to change the u+02BC into u+0027 in the page contents, but I haven't figured out how to change it in the page title (how to create a pywikibot script to change the apostroph in the page titles).

Unfortunately despite warning from me, our contributors often forget to check their apostroph before creating a wiki page.

Nerdspeak: The technical issue here is that we have ICU normalization enabled by default everywhere that we can (though not in monolithic analyzers), but ICU folding does the mapping from curly quotes to straight quotes. ICU folding is not enabled by default because it is too aggressive and needs to be customized for each language/alphabet/writing system to keep it from causing problems.

This is in our work queue now, so I will get to it in the normal course of things (it'll be a little while, though). I may do some sort of mini hack for Nias/nia and open a new ticket for the larger task of addressing this consistently everywhere. I'm currently leaning toward a new focused apostrophe filter (apostrophe_norm, maybe) that converts curly quotes (‘ & ’) and maybe backticks/backquotes/grave accents (`) to straight apostrophes (')—but to do so widely by defaultwill require more careful testing.

Clearly Nias Wikipedia needs this fix, though, since they have at least one pair of titles that differ only by the curliness of the quotes: Hili'adulo and Hili’adulo. You don't need to be able to understand the language to see that those are very similar and probably should be the same article/disambiguation page. I'll probably set up apostrophe_norm just for nia and open a ticket to enable it everywhere by default (and maybe look into aggressive_splitting for English at the same time.. enwiki currently would index both Hili'adulo and Hili’adulo as two tokens: hili and adulo.

In the meantime, a couple of questions and comments:

• @Sannita, you mentioned diacritics in the title of the ticket, but I don't see any discussed in the ticket. Are there any other specific diacritics that are causing trouble? If so, I can probably set up a nia-specific version of ICU folding to handle them.
• @Aklapper the results you see from the completion suggester with a backtick aren't really processing the backtick as anything other than a punctuation character that it can ignore or replace with another character to get a good match. You get the same results if you use %, (, !, or @. (Other punctuation gets treated a bit differently. Using - or [ gets slightly different results. The point is that the backtick isn't being treated in any special way—good or bad—because it is similar to an apostrophe.)

@TJones This is a question for @Slaia, I only wrote the ticket in their stance to help them.

Hallo @TJones The issue with diacritic seems gone. I can't replicate the issue I had a few weeks ago. In the past the word u'o'ö-o'ö'ö (I'm following you) would not show the result where u'o'ö (I follow) is also a part of the word or sentence. Thanks.

(Updated title to reflect focus on apostrophes after discussion here.)

Change 822647 had a related patch set uploaded (by EJoseph; author: EJoseph):

[mediawiki/extensions/CirrusSearch@master] Add apostrophe filter to the character filter nia wiki

https://gerrit.wikimedia.org/r/822647

Change 822647 merged by jenkins-bot:

[mediawiki/extensions/CirrusSearch@master] Add apostrophe filter to the character filter nia wiki

https://gerrit.wikimedia.org/r/822647

FYI: The reindexing to enable the new apostrophe handling is complete. I checked and searching for Hili'adulo, Hili’adulo, or Hili‘adulo all return the same 66 results.

Follow up task to look at this more broadly: T315118: Handle variation in apostrophe-like characters better