Sudachi analysis chain fails on long emoji sequence
Closed, ResolvedPublic3 Estimated Story Points
Actions

Assigned To

Authored By

	EBernhardson
	Aug 18 2025, 7:54 PM

Description

Attempting to reindex jawiki_general fails with:

{
  "index": "jawiki_general_1755293360",
  "type": "_doc",
  "id": "5112606",
  "cause": {
    "type": "execution_exception",
    "reason": "execution_exception: java.lang.IndexOutOfBoundsException: Index -1024 out of bounds for length 87503",
    "caused_by": {
      "type": "index_out_of_bounds_exception",
      "reason": "index_out_of_bounds_exception: Index -1024 out of bounds for length 87503"
    }
  },
  "status": 500
}

The referenced page contains essentially the same emoji repeated many times:

https://ja.wikipedia.org/wiki/%E5%88%A9%E7%94%A8%E8%80%85:%E4%B8%87%E6%AD%B3%E5%B8%9D%E5%9B%BD

We need to resolve whatever problem is going on there so that jawiki_general can pick up the updated analysis chain. This also means that content changes on wikis that use the japanese analysis chain may fail the indexing pipeline and be un-updatable.

Details

Related Changes in Gerrit:

	Subject	Repo	Branch	Lines +/-
	Break long sequences of characters for Sudachi	mediawiki/extensions/CirrusSearch	master	+57 -11

Customize query in gerrit

Related Objects

Mentioned In: T402629: Monitor CirrusSearch index failures
T317599: Allow ^ and $ in intitle regex search

Event Timeline

EBernhardson created this task.Aug 18 2025, 7:54 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 18 2025, 7:54 PM

bking subscribed.Aug 18 2025, 8:19 PM

TJones changed the task status from Open to In Progress.Aug 18 2025, 10:38 PM

EBernhardson mentioned this in T317599: Allow ^ and $ in intitle regex search.Aug 19 2025, 3:02 PM

Change #1180192 had a related patch set uploaded (by Tjones; author: Tjones):

[mediawiki/extensions/CirrusSearch@master] Break long sequences of characters for Sudachi

https://gerrit.wikimedia.org/r/1180192

gerritbot added a project: Patch-For-Review.Aug 19 2025, 6:36 PM

I can generate the error with the right sequences of katakana, hiragana, emoji, Gothic, and Thai characters.. long sequences of multibyte characters seems to be a reliable source of errors. I never had errors under 8000 characters. Sometimes I got errors at 16000 characters, sometimes 32000 (depending on character type). It looks like the threshold for problems is often around 11,000 characters, but I didn't dig too deeply.

Oddly, 64000-character tokens reliably get broken up into blocks of 4096 characters (but 16000 and 32000 don't). I guess there some a threshold of clearly too long that triggers the chunking.

I suspect threre's a mismatch between counting characters and counting bytes, and/or problems with the OpenSearch-internal representations of multibyte characters (🤓 → \uD83E\uDD13, 𐌳 → \uD800\uDF33, etc.).

I've added a (hacky) char filter to look for sequences of over 8000 characters without a reliable Sudachi word boundary (\s or certain CJK punctuation characters) and add a space after those 8000 characters.

The longest sequences I could find in the wild (on jawiki and jawikisource) were in the ~800 character range (and those do all get broken up into normal, mostly 1–4 character tokens by the tokenizer), so 8000 should let almost all natural text go through unharmed.

A patch is up for review, I'll add some more details to my Sudachi notes on MediaWiki, and I'll open a ticket upstream.

TJones moved this task from Incoming to Needs Review on the Discovery-Search (2025.07.25 - 2025.08.15) board.Aug 19 2025, 6:55 PM

Change #1180192 merged by jenkins-bot:

[mediawiki/extensions/CirrusSearch@master] Break long sequences of characters for Sudachi

https://gerrit.wikimedia.org/r/1180192

Maintenance_bot removed a project: Patch-For-Review.Aug 20 2025, 7:30 AM

ReleaseTaggerBot added a project: MW-1.45-notes (1.45.0-wmf.16; 2025-08-26).Aug 20 2025, 8:00 AM

pfischer edited projects, added Discovery-Search (2025.08.15 - 2025.09.05); removed Discovery-Search (2025.07.25 - 2025.08.15).Aug 20 2025, 2:38 PM

pfischer moved this task from Incoming to Needs Review on the Discovery-Search (2025.08.15 - 2025.09.05) board.

TJones moved this task from Needs Review to To be Deployed on the Discovery-Search (2025.08.15 - 2025.09.05) board.Aug 20 2025, 3:47 PM

TJones triaged this task as High priority.Aug 21 2025, 5:24 PM

TJones set the point value for this task to 3.