It looks like this change could be the cause: https://gerrit.wikimedia.org/r/c/mediawiki/vendor/+/1057922

It's from this patch, yes,
https://gerrit.wikimedia.org/r/c/mediawiki/services/parsoid/+/540620/8

but the difference is going from substr to PHPUtils::safeSubstr which adds the assertion. In other words, it's likely surfacing the issue, rather than being the cause.

The root cause is usually old revisions with bad UTF-8, so the volume tends to vary based on how many folks are digging through back history.

I'll take a look to see if I can narrow things down more specifically, but as @arlo says this isn't a new issue, just a new log.

I'll take a look to see if I can narrow things down more specifically, but as @arlo says this isn't a new issue, just a new log.

Would it be possible to add more specific information about the error to the logged message field?

The data in the log entry is pretty generic: message is "Invariant failed: Bad UTF-8 at start of string", source file is "assert/src/Assert.php", etc. The only place with discriminating info is the stacktrace itself, but that field is not indexed and we cannot add filters for it on the dashboards. So the errors are creating a significant level of noise in the logs that we can't filter out at the moment or more precisely, I'm worried about filtering out too much by accident.

I'm having trouble reproducing the stack trace in the example; it's a html-to-wikitext conversion driven by an edit in Visual Editor. I suspect the issue is the following heading:

== أعراض الألم النفسي<ref>{{استشهاد ويب|عنوان=الألم النفسي: أسبابه، وأعراضه، وعلاجه|مسار=https://www.annajah.net/%D8%A7%D9%84%D8%A3%D9%84%D9%85-%D8%A7%D9%84%D9%86%D9%81%D8%B3%D9%8A-%D8%A3%D8%B3%D8%A8%D8%A7%D8%A8%D9%87-%D9%88%D8%A3%D8%B9%D8%B1%D8%A7%D8%B6%D9%87-%D9%88%D8%B9%D9%84%D8%A7%D8%AC%D9%87-article-33779|تاريخ الوصول=2024-08-06|صحيفة=النجاح|لغة=ar-SA}}</ref><ref>{{استشهاد ويب|عنوان=الألم النفسي وكيفية التعامل معه|مسار=https://altibbi.com/%D9%85%D9%82%D8%A7%D9%84%D8%A7%D8%AA-%D8%B7%D8%A8%D9%8A%D8%A9/%D8%A7%D9%84%D8%B5%D8%AD%D8%A9-%D8%A7%D9%84%D9%86%D9%81%D8%B3%D9%8A%D8%A9/%D8%A7%D9%84%D8%A7%D9%84%D9%85-%D8%A7%D9%84%D9%86%D9%81%D8%B3%D9%8A-9171|تاريخ الوصول=2024-08-06|صحيفة=الطبي|مؤلف=Altibbi.com|لغة=ar-US}}</ref> ==

which is quite an impressive mess.

In T371875#10050644, @jnuche wrote:

I'll take a look to see if I can narrow things down more specifically, but as @arlo says this isn't a new issue, just a new log.

Would it be possible to add more specific information about the error to the logged message field?

The data in the log entry is pretty generic: message is "Invariant failed: Bad UTF-8 at start of string", source file is "assert/src/Assert.php", etc. The only place with discriminating info is the stacktrace itself, but that field is not indexed and we cannot add filters for it on the dashboards. So the errors are creating a significant level of noise in the logs that we can't filter out at the moment or more precisely, I'm worried about filtering out too much by accident.

I'd suggest filtering on the url field, /w/api.php, which also helps isolate "problems we can realistically debug" (ie, things which come from a page render) from "things we can't realistically debug" (ie, UTF-8 issues stemming from bad user input).

I'd suggest filtering on the url field, /w/api.php, which also helps isolate "problems we can realistically debug" (ie, things which come from a page render) from "things we can't realistically debug" (ie, UTF-8 issues stemming from bad user input).

That sounds good, it will at least limit the filters to Parsoid calls.

@cscott thanks a lot for taking a look, it sounds like we are reasonably confident nothing has changed other than the logging itself. So I'm going to make the call and roll the train forward again.

About 9150 entries in the last 2 weeks ... about 1 every 2 minutes which is higher than I would like. I am not convinced this is actually bad UTF in revisions. This is across a number of different wikis (jawiki contributing almost 45% of the errors).

But, I am having a hard time reproducing this with some random edits on some of the pages in the logs. Need to figure out a different strategy and/or dump more logs since this is clearly user-activity triggered (edits).

Digging into the exception traces a bit more, it looks like these 4 lines account for pretty much every exception (except 7):

• "Separators.php(899)"
• "Separators.php(902)"
• "Separators.php(907)"
• "Separators.php(642)"

Of those the last one has only 7 instances as well. So, it is really the first three we should focus on first. Those all construct new DSR ranges by doing offset arithmetic by looking at DSR values of two nodes. So, that is a clue to inspect more and if the code guarantees that those constructed offsets are always guaranteed to yield valid UTF8 offsets or if the code needs to have other checks in place to avoid these exception scenarios. I think the code there tries to make best-faith efforts to infer a separator string and rejects the string if it is not a valid separator - so any invalid offsets should not cause exceptions but instead feed the valid separator check. We'll take the followup investigation off-phan now and that might yield a patch.

Change #1062776 had a related patch set uploaded (by Subramanya Sastry; author: Subramanya Sastry):

[mediawiki/services/parsoid@master] WIP: html2wt: Add option to skip utf8-validity checks

https://gerrit.wikimedia.org/r/1062776

Change #1063059 had a related patch set uploaded (by C. Scott Ananian; author: Subramanya Sastry):

[mediawiki/services/parsoid@master] Html2Wt: don't construct substrings with bad UTF-8

https://gerrit.wikimedia.org/r/1063059

Change #1063060 had a related patch set uploaded (by C. Scott Ananian; author: C. Scott Ananian):

[mediawiki/services/parsoid@master] Html2wt: Fix string creation with bad UTF-8 by validating offsets in DSRs

https://gerrit.wikimedia.org/r/1063060

Change #1063059 merged by jenkins-bot:

[mediawiki/services/parsoid@master] Html2Wt: don't construct substrings with bad UTF-8

https://gerrit.wikimedia.org/r/1063059

Change #1062776 merged by jenkins-bot:

[mediawiki/services/parsoid@master] Html2Wt: Remove substring UTF-8 checks where they are unsafe

https://gerrit.wikimedia.org/r/1062776

Change #1063811 had a related patch set uploaded (by Subramanya Sastry; author: Subramanya Sastry):

[mediawiki/vendor@master] Bump wikimedia/parsoid to 0.20.0-a17

https://gerrit.wikimedia.org/r/1063811

Change #1063811 merged by jenkins-bot:

[mediawiki/vendor@master] Bump wikimedia/parsoid to 0.20.0-a17

https://gerrit.wikimedia.org/r/1063811

Change #1063060 merged by jenkins-bot:

[mediawiki/services/parsoid@master] Html2wt: Fix string creation with bad UTF-8 by validating offsets in DSRs

https://gerrit.wikimedia.org/r/1063060

Change #1066683 had a related patch set uploaded (by Isabelle Hurbain-Palatin; author: Isabelle Hurbain-Palatin):

[mediawiki/vendor@master] Bump wikimedia/parsoid to 0.20.0-a18

https://gerrit.wikimedia.org/r/1066683

Change #1066683 merged by jenkins-bot:

[mediawiki/vendor@master] Bump wikimedia/parsoid to 0.20.0-a18

https://gerrit.wikimedia.org/r/1066683