Refactor slow global analysis components
Closed, ResolvedPublic13 Estimated Story Points
Actions

Assigned To

Authored By

	TJones
	Sep 11 2023, 3:16 PM

Description

T342444 was halted because the reindexing was too much slow.

Update config with a more efficient interim analysis chain in case any reindexing needs to be done.
Refactor recent analysis upgrades (acronyms and camelCase) to be acceptably efficient as custom filters in ~~the extra plugin~~ a new plugin
- ~~Enable plugin version-checking in analysis config (so we know we have the new extra plugin)~~
- Enable less expensive fallback versions of camelCase and acronym processing for 3rd party users without the new plugin
Possibly investigate other slow points in global configs (implement immediately or open new tickets)

New dependency: We can/should link this with T332337, which also needs a new filter and put everything in one new plugin.

Details

Subject	Repo	Branch	Lines +/-
Add limited_mapping to Textify Plugin	search/extra	master	+569 -2
Refactor CamelCase Analysis into Textify Plugin	search/extra	master	+398 -4
Refactor Acronym Fixer Analysis into New Textify Plugin	search/extra	master	+814 -8
Allow limited_mapping when textify plugin is present	mediawiki/extensions/CirrusSearch	master	+859 -825
Config Acronym Fixer Plugin	mediawiki/extensions/CirrusSearch	master	+817 -3
Allow Fallback Filters, Config CamelCase Plugin	mediawiki/extensions/CirrusSearch	master	+1 K -466
Refactor and Revert Analysis Harmonization	mediawiki/extensions/CirrusSearch	master	+196 -989

Customize query in gerrit

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Open		None	T219550 [EPIC] Harmonize language analysis across languages
		Resolved		TJones	T346051 Refactor slow global analysis components

Event Timeline

TJones created this task.Sep 11 2023, 3:16 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSep 11 2023, 3:16 PM

TJones updated the task description. (Show Details)Sep 11 2023, 3:19 PM

TJones updated the task description. (Show Details)

TJones renamed this task from Refactor slow analysis components to Refactor slow global analysis components.Sep 11 2023, 3:21 PM

Gehel moved this task from needs triage to Current work on the Discovery-Search board.Sep 11 2023, 3:22 PM

Gehel edited projects, added Discovery-Search (Current work); removed Discovery-Search.

TJones claimed this task.Sep 11 2023, 3:28 PM

Gehel set the point value for this task to 13.Sep 11 2023, 3:39 PM

Gehel moved this task from Incoming to Ready for Dev -- SWE on the Discovery-Search (Current work) board.

TJones moved this task from Ready for Dev -- SWE to In Progress on the Discovery-Search (Current work) board.Sep 11 2023, 3:40 PM

TJones mentioned this in T332342: Standardize ASCII-folding/ICU-folding across analyzers.

Change 957806 had a related patch set uploaded (by Tjones; author: Tjones):

[mediawiki/extensions/CirrusSearch@master] Refactor and Revert Analysis Harmonization

https://gerrit.wikimedia.org/r/957806

gerritbot added a project: Patch-For-Review.Sep 14 2023, 7:29 PM

Change 957806 merged by jenkins-bot:

[mediawiki/extensions/CirrusSearch@master] Refactor and Revert Analysis Harmonization

https://gerrit.wikimedia.org/r/957806

Maintenance_bot removed a project: Patch-For-Review.Sep 15 2023, 9:10 PM

TJones updated the task description. (Show Details)Sep 21 2023, 9:54 PM

TJones updated the task description. (Show Details)Sep 27 2023, 4:47 PM

We previously discussed how to bundle the new filters, but talked about it again today.

Since acronym and camelCase processing aren't language-specific, creating a separate plugin isn't an obvious requirement or even desirable. Moving them into the extra plugin made sense from an architectural point of view, but the added complexity for our own deployment, 3rd party users, and even developers is undesirable. Trying to resolve everything in the config builder by testing for specific WMF versions of plugins (e.g., "v7.10.2-wmf5 or newer") is possibly more complexity than it is worth at the moment.

OTOH, creating and checking for the presence of a new plugin is easy. Though it is possible that in the future that the overhead of many plugins is a problem, but at the moment there is no evidence of that. For now, our standard operating procedure will be to create a new plugin when we have a batch of new filters to create.

As a big-picture compromise, it makes sense to work on T332337 (ICU tokenizer repair) before returning to T332342 (folding), and bundle the new filter there with the two here, so that all three new filters can be in one plugin.

Change 965602 had a related patch set uploaded (by Tjones; author: Tjones):

[search/extra@master] Refactor Acronym Fixer Analysis into New Textify Plugin

https://gerrit.wikimedia.org/r/965602

Change 965603 had a related patch set uploaded (by Tjones; author: Tjones):

[search/extra@master] Refactor CamelCase Analysis into Textify Plugin

https://gerrit.wikimedia.org/r/965603

Change 965793 had a related patch set uploaded (by Tjones; author: Tjones):

[search/extra@master] Add limited_mapping to Textify Plugin

https://gerrit.wikimedia.org/r/965793

TJones updated the task description. (Show Details)Oct 13 2023, 7:21 PM

TJones updated the task description. (Show Details)Oct 13 2023, 7:28 PM

Change 965575 had a related patch set uploaded (by Tjones; author: Tjones):

[mediawiki/extensions/CirrusSearch@master] Allow Fallback Filters, Config CamelCase Plugin

https://gerrit.wikimedia.org/r/965575

Change 965576 had a related patch set uploaded (by Tjones; author: Tjones):

[mediawiki/extensions/CirrusSearch@master] Config Acronym Fixer Plugin

https://gerrit.wikimedia.org/r/965576

Gehel added a parent task: T219550: [EPIC] Harmonize language analysis across languages.Oct 20 2023, 1:37 PM

Change 967912 had a related patch set uploaded (by Tjones; author: Tjones):

[mediawiki/extensions/CirrusSearch@master] Allow limited_mapping when textify plugin is present

https://gerrit.wikimedia.org/r/967912

TJones moved this task from In Progress to Needs review on the Discovery-Search (Current work) board.Oct 23 2023, 8:14 PM

Change 965575 merged by jenkins-bot:

[mediawiki/extensions/CirrusSearch@master] Allow Fallback Filters, Config CamelCase Plugin

https://gerrit.wikimedia.org/r/965575

Change 965576 merged by Tjones:

[mediawiki/extensions/CirrusSearch@master] Config Acronym Fixer Plugin

https://gerrit.wikimedia.org/r/965576

Dev notes and details on Mediawiki.

Highlights:

Acronym extra load time decreased from 274.1% to 4.4%!
camelCase extra load time decreased from 19.9% to 3.6%
limited_mappings are ~50% faster then regular mappings for simple one-char to one-char mappings
I'm going to work on T332337: Repair multi-script tokens split by the ICU tokenizer next instead of going back to T332342: Standardize ASCII-folding/ICU-folding across analyzers to roll everything into one new plugin.