We previously discussed how to bundle the new filters, but talked about it again today.

Moved back to ready for dev while working on T346051

This has been deployed, but the reindexing ws stopped for being too slow. I'll move this ticket into needs reporting and open a new one for the new efficiency refactor.

Sounds like my local reindexing is insufficient for detecting non-egregious slow downs in indexing speed. (I know I have other overhead—I guess it's even more than I thought.) Should we pause the reindex and investigate more thoroughly on RelForge, with the possibility of reverting some changes after finding the slowest ones?

acronym_fixer is rather complicated, as expected. word_break_helper is a little complicated, unexpectedly! More on MediaWiki.

While harmonizing, I noticed that the Hebrew analysis chain was creating a lot of duplicate tokens. Adding a remove_duplicates filter removed 19.7% (Wikipedia) to 22.7% (Wiktionary) of all tokens—all non-Hebrew and many Hebrew tokens were duplicated! Did a lot of refactoring (checked off the task above!), too.

I'll do a write up of the before-and-after impact of the reindexing and post a link here, but anyone can do the reindexing and finish the ticket without that.

@Pols12, the code is deployed, but not activated yet. In our workflow, we generally close tickets when the code is deployed, separate from when the feature is available.

I'm going to close this one because we've deprecated aggressive_splitting as too aggressive for the text field. It is still used in the short_text field, but the context there is constrained and not language-specific, so it's less likely to accidentally get messy.

If we can get a list of mappings, this should be technically straightforward. I will review the lists provided, and consult with a linguist I know who lives in Mongolia to see if I missed anything else obvious.

I think I found the input that causes the problem.

My full writeup is on Mediawiki.

Sorry for the hokey pokey—you put the ticket in, you take the ticket out.. you put the ticket in, and you shake it all about—but the aggressive_splitting ticket (T219108) overlaps with this one too much. And! I discovered I can do what I want for acronym collapsing with a regex (probably.. still checking on details) rather than a custom filter, which makes this easier—and I'd feel better about deploying word_break_helper everywhere with that fix in place.

More details to come, but aggressive_splitting (which is a word_delimiter filter underneath) is just too aggressive. It breaks things ICU normalization does better, and word_break_helper (T170625) does or can do the good things aggressive_splitting does. My new plan is to deactivate aggressive_splitting on English-language wikis and replace it with a split_camelCase filter that addresses the original issue of "FilesystemHierarchyStandard" in this ticket, and delegate the good things it does to word_break_helper.

Full write up on MediaWiki.

Holy Guacamole, Batman! It's all done!

Only looking at changes from the better apostrophe handling, because everything else got reindexed as part of reindexing all wikis for another project before the Turkish plugin was deployed. At least we tested the fallback config for when the Turkish plugin is missing, and it worked just fine!

• No meaningful changes to ZRR or top result for this sample of queries.
• Good changes to queries getting both more results (improved recall—Italian oggetto matching all'oggetto) and fewer results (improved precision—l no longer matches l'administration).

In T332355#8860855, @RKemper wrote:

We've built the new package 7.10.2-5. Haven't yet done a restart of hosts.

• Estonian got a new stemmer, and it had a pretty big impact! 1 in 6 previous Estonian Wikipedia zero-results queries get results. Almost 1 in 3 of non-zero-results queries get more results, and more than 1 in 8 queries had their top result change.

The new Estonian analyzer looks good, and enabling it has a big impact:

• 4.372% of Wiktionary tokens (532 distinct, including case variants) and 13.146% of Wikipedia tokens (987 distinct) were filtered as stop words.
• The merges from stemming were quite significant (even more so for Wikipedia):
  • Estonian Wiktionary: 11,044 tokens (6.912% of tokens) were merged into 2,487 groups (2.895% of groups)
  • Estonian Wikipedia: 497,501 tokens (22.675% of tokens) were merged into 23,075 groups (7.373% of groups)

A review of the cirrus-settings-dump and a test query show things are deployed as expected.

@EBernhardson, it is working! I also verified some of the original examples from the Wikidata discussion page.