After thinking about this more while writing up the description, and talking with @dcausse today, I think I'm going to close this ticket for now. If this shows up more often and more annoyingly than it seems to me that it will, someone can re-open the ticket.

Looking at the en/fr/he report and the 20-wiki report.

@Bugreporter, the code you linked to is currently only used for as-you-type suggestions, but your mention of page contents sounds like you are looking for matching in fulltext search results, too. Are thinking of one, the other, or both?

@Pppery, thanks for your bolder edits. I'm always a little hesitant to jump in and make sweeping changes from my WMF account. I appreciate you taking the info I was able to provide and editing it into shape! If you feel like things are done, we can move this ticket to "Done" on the Discovery workboard.

I also added a blurb to next week's Tech News.

Okay, I think I'm done with the updates. (Thanks to @Shirayuki for marking them for translation and making a few other updates, too.)

In T375215#11426047, @stjn wrote:

Is there any plan to make it so that search pages like https://ru.wikipedia.org/wiki/Special:Search/,fhfr_j,fvf («Барак Обама», Barack Obama) display relevant results as well?

I've made some edits and added some information. I will try to finish reviewing the rest of the section tomorrow to soften the language throughout from "never do use a bare regex" to "try no to use bare regexes". (Though if someone beats me to it, I won't complain!)

We're off-topic for this ticket, but I'll reply to these here. @dcausse, do you have the link for the on-wiki help with regex searches you mentioned?

Started looking at this, and it's a little harder to match the style and tone than I expected.

In T410758#11399135, @Taylor wrote:

Indeed it is documented. But how to search for

||}}{{de|

on a wiki where "de" is a common preposition or pronoun? The stupid indexed search will ignore everything except the "de", and since "de" is on every page, it will not generate any "domain".

Maybe wikis would need a simple plain-text search, without all the flawed indexed "magic", and without the resource-hoggy regex magic.

The plugins have been deployed to production CirrusSearch.

Added this:

The Wikimedia Search Team has recreated the "DWIM" gadget functionality server-side, for Russian and Hebrew Wikipedias. This feature appends cross-keyboard suggestions to the standard search-box suggestions. For example, searching for cxfcnmt on Russian Wikipedia will now append suggestions for счастье ("happiness"). We plan to enable this feature for other Russian and Hebrew wikis next week. See also Phabricator T408734 and related tickets.

In T408734#11378210, @dcausse wrote:

could you draft a message for the tech news ...

Moving this to triage, since it is the next step in the NORM/SecondTry project.

Mediawiki notes.

Once this is live, we'll need to reindex. (Added to T408431)

I often open the target folder to find the jar so I can deploy it locally. I also use the Finder with Java projects to recursively open folders to get all the way to the bottom quickly. (Anything that relies on people not opening folders is doomed to fail eventually.)

I believe we should find what is causing them to appear in the first place

As part of T407440: Regexes with four 32-bit characters throw errors I've made some changes to the opensearch-extra plugin which also needs to be deployed. Would it make sense to wait and do them both at once?

Thanks for running all the individual language reports—very interesting to look at them all. I skimmed them, comparing the charts for each to the all_wikis charts.

One little spot in the code was counting characters instead of codepoints, so the last matched four-byte character got split in half by the closing highlight. Somewhere up the chain the unmatched high and low surrogates got stripped and the character disappeared.

I've been poking around at this a bit, and it definitely seems to be the Experimental Highlighter that's causing the problem. When I disable it locally, highlighting works fine.

Patch is up, updated notes are done (new section here). Generally, there were a few hiccups, but only in cases where we are already limited to making a best-guess, so the deviations from the previous algorithm are acceptable.

In-progress notes (after some vacation—yay!—and illness—ugh!) on MediaWiki. I think I've got a workable algorithm, but I need to do a bit more implementation and integration.

I also thought at first that it might be treating [𐌰𐌹𐌶] as a set of six surrogates rather than three characters, but even without the character class, you get the funky missing highlight: intitle:"𐌰𐌹𐌶" insource:/𐌰/

@Gehel, we use the icu_tokenizer (or our version of it) almost everywhere, and it does split on lots of hyphens and dashes correctly. I tested a bunch of dash-like symbols (- ‐ ‑ ﹣ － ‒ – — ゠ ⹀), and tested all of the tokenizers we use: ICU, standard, smartCN, Hebrew, sudachi, Kuromoji, Nori, Thai.

default_1 looks like a winner if default_1_variant turns out to be too expensive. The A/B test results on latency and clicks will be interesting.

The variant option really is doing the heavy lifting.

Nice write up! Your conclusions make sense to me. Do you want to do an A/B/C/D test, or a series of (three?) A/B tests?

@TheDJ, this seems to be working for the current default skin. Could we either close the ticket (under the theory that not every old skin must be upgraded to have every new feature) or change the title and description to reflect which skins need to be fixed (under the theory that Vector isn't that old)? In the second case, Monobook has the dead key problem, too.

The example file no longer has the information template in it two times, and regex results look reasonable. If there is another example, please re-open or create a new ticket.

In the intervening time, this has been taken care of by the icu_normalize filter which is enabled everywhere.

A single multi-script mapping turns out not to be so great. While most letters map one-to-one across keyboards (я ↔︎ z), punctuation does not, and there are chains of mappings. Those in RU/QWERTY:

I can generate the error with the right sequences of katakana, hiragana, emoji, Gothic, and Thai characters.. long sequences of multibyte characters seems to be a reliable source of errors. I never had errors under 8000 characters. Sometimes I got errors at 16000 characters, sometimes 32000 (depending on character type). It looks like the threshold for problems is often around 11,000 characters, but I didn't dig too deeply.