Page MenuHomePhabricator

Regexes with four 32-bit characters throw errors
Open, Needs TriagePublic5 Estimated Story PointsBUG REPORT

Description

While working on T404632, I noticed that insource:/๐Œ€๐Œ๐Œ‚๐Œƒ/ fails with an error, regardless of whether it matches anything. Every regex I've tried with with 4 or more four-byte characters fails. They also fail on my local wiki which has a fix for the highlighting issue in T404632. I tried the regexes on enwiki, frwiki, dewiki, jawikt, and commons Special Search.

Erik pulled out this message:

Provided analyzer generated more than one token, if using 3grams make sure to use a 3grams analyzer, for input [\uDF00\uD800\uDF01\uD800\uDF02\uD800\uDF03] first is [\uDF00\uD800\uDF01\uD800\uDF02] but [\uD800\uDF01\uD800\uDF02\uD800\uDF03] was generated.

A little more digging reveals that a regex with a four-byte character followed by three or more characters of any kind causes the error. (e.g., insource:/๐Œ€aaa/)

(Based on the message Erik found and the behavior I'm seeing, my guess is that the trigram extractor is stepping one codepoint forward and not one character when it moves to get the "next" trigram... but we will have to see.)

Details

Related Changes in Gerrit:

Event Timeline

Restricted Application added a subscriber: Aklapper. ยท View Herald TranscriptOct 15 2025, 9:18 PM
pfischer set the point value for this task to 5.Oct 20 2025, 3:38 PM

Change #1198379 had a related patch set uploaded (by Tjones; author: Tjones):

[search/extra@master] Fix bug in multi-codepoint trigram parsing

https://gerrit.wikimedia.org/r/1198379

Change #1198379 merged by jenkins-bot:

[search/extra@master] Fix bug in multi-codepoint trigram parsing

https://gerrit.wikimedia.org/r/1198379