Page MenuHomePhabricator
Create Task
Maniphest T371617

Wikimedia\Assert\InvariantException: Invariant failed: Bad UTF-8 at end of string (2 byte sequence)
Open, Needs TriagePublicPRODUCTION ERROR
Actions

Description

Error
labels.normalized_message
[{reqId}] {exception_url}   Wikimedia\Assert\InvariantException: Invariant failed: Bad UTF-8 at end of string (2 byte sequence)
error.stack_trace
from /srv/mediawiki/php-1.43.0-wmf.15/vendor/wikimedia/assert/src/Assert.php(231)
#0 /srv/mediawiki/php-1.43.0-wmf.15/vendor/wikimedia/parsoid/src/Utils/PHPUtils.php(194): Wikimedia\Assert\Assert::invariant(boolean, string)
#1 /srv/mediawiki/php-1.43.0-wmf.15/vendor/wikimedia/parsoid/src/Wt2Html/DOM/Processors/DOMRangeBuilder.php(956): Wikimedia\Parsoid\Utils\PHPUtils::safeSubstr(string, integer, integer)
#2 /srv/mediawiki/php-1.43.0-wmf.15/vendor/wikimedia/parsoid/src/Wt2Html/DOM/Processors/DOMRangeBuilder.php(1270): Wikimedia\Parsoid\Wt2Html\DOM\Processors\DOMRangeBuilder->encapsulateTemplates(array)
#3 /srv/mediawiki/php-1.43.0-wmf.15/vendor/wikimedia/parsoid/src/Wt2Html/DOM/Processors/WrapTemplates.php(21): Wikimedia\Parsoid\Wt2Html\DOM\Processors\DOMRangeBuilder->execute(Wikimedia\Parsoid\DOM\Element)
#4 /srv/mediawiki/php-1.43.0-wmf.15/vendor/wikimedia/parsoid/src/Wt2Html/DOMPostProcessor.php(139): Wikimedia\Parsoid\Wt2Html\DOM\Processors\WrapTemplates->run(Wikimedia\Parsoid\Config\Env, Wikimedia\Parsoid\DOM\Element, array, boolean)
#5 /srv/mediawiki/php-1.43.0-wmf.15/vendor/wikimedia/parsoid/src/Wt2Html/DOMPostProcessor.php(903): Wikimedia\Parsoid\Wt2Html\DOMPostProcessor->Wikimedia\Parsoid\Wt2Html\{closure}(Wikimedia\Parsoid\DOM\Element, array, boolean)
#6 /srv/mediawiki/php-1.43.0-wmf.15/vendor/wikimedia/parsoid/src/Wt2Html/DOMPostProcessor.php(944): Wikimedia\Parsoid\Wt2Html\DOMPostProcessor->doPostProcess(Wikimedia\Parsoid\DOM\Element)
#7 /srv/mediawiki/php-1.43.0-wmf.15/vendor/wikimedia/parsoid/src/Wt2Html/DOMPostProcessor.php(962): Wikimedia\Parsoid\Wt2Html\DOMPostProcessor->process(Wikimedia\Parsoid\DOM\Element)
#8 /srv/mediawiki/php-1.43.0-wmf.15/vendor/wikimedia/parsoid/src/Wt2Html/ParserPipeline.php(170): Wikimedia\Parsoid\Wt2Html\DOMPostProcessor->processChunkily(string, array)
#9 /srv/mediawiki/php-1.43.0-wmf.15/vendor/wikimedia/parsoid/src/Wt2Html/ParserPipelineFactory.php(279): Wikimedia\Parsoid\Wt2Html\ParserPipeline->parseChunkily(string, array)
#10 /srv/mediawiki/php-1.43.0-wmf.15/vendor/wikimedia/parsoid/src/Wikitext/ContentModelHandler.php(159): Wikimedia\Parsoid\Wt2Html\ParserPipelineFactory->parse(string)
#11 /srv/mediawiki/php-1.43.0-wmf.15/vendor/wikimedia/parsoid/src/Parsoid.php(194): Wikimedia\Parsoid\Wikitext\ContentModelHandler->toDOM(Wikimedia\Parsoid\Ext\ParsoidExtensionAPI)
#12 /srv/mediawiki/php-1.43.0-wmf.15/vendor/wikimedia/parsoid/src/Parsoid.php(236): Wikimedia\Parsoid\Parsoid->parseWikitext(MediaWiki\Parser\Parsoid\Config\PageConfig, MediaWiki\Parser\ParserOutput, array)
#13 /srv/mediawiki/php-1.43.0-wmf.15/includes/parser/Parsoid/ParsoidParser.php(149): Wikimedia\Parsoid\Parsoid->wikitext2html(MediaWiki\Parser\Parsoid\Config\PageConfig, array, NULL, MediaWiki\Parser\ParserOutput)
#14 /srv/mediawiki/php-1.43.0-wmf.15/includes/parser/Parsoid/ParsoidParser.php(287): MediaWiki\Parser\Parsoid\ParsoidParser->genParserOutput(MediaWiki\Parser\Parsoid\Config\PageConfig, ParserOptions)
#15 /srv/mediawiki/php-1.43.0-wmf.15/includes/Rest/Handler/Helper/HtmlOutputRendererHelper.php(931): MediaWiki\Parser\Parsoid\ParsoidParser->parseFakeRevision(MediaWiki\Revision\MutableRevisionRecord, MediaWiki\Page\PageStoreRecord, ParserOptions)
#16 /srv/mediawiki/php-1.43.0-wmf.15/includes/Rest/Handler/Helper/HtmlOutputRendererHelper.php(894): MediaWiki\Rest\Handler\Helper\HtmlOutputRendererHelper->parseUncacheable(MediaWiki\Page\PageStoreRecord, ParserOptions, MediaWiki\Revision\MutableRevisionRecord, boolean)
#17 /srv/mediawiki/php-1.43.0-wmf.15/includes/Rest/Handler/Helper/HtmlOutputRendererHelper.php(622): MediaWiki\Rest\Handler\Helper\HtmlOutputRendererHelper->getParserOutputInternal(ParserOptions)
#18 /srv/mediawiki/php-1.43.0-wmf.15/includes/Rest/Handler/Helper/HtmlOutputRendererHelper.php(717): MediaWiki\Rest\Handler\Helper\HtmlOutputRendererHelper->getParserOutput()
#19 /srv/mediawiki/php-1.43.0-wmf.15/includes/Rest/Handler/ParsoidHandler.php(736): MediaWiki\Rest\Handler\Helper\HtmlOutputRendererHelper->getPageBundle()
#20 /srv/mediawiki/php-1.43.0-wmf.15/includes/Rest/Handler/TransformHandler.php(136): MediaWiki\Rest\Handler\ParsoidHandler->wt2html(MediaWiki\Parser\Parsoid\Config\PageConfig, array, string)
#21 /srv/mediawiki/php-1.43.0-wmf.15/includes/Rest/Module/Module.php(380): MediaWiki\Rest\Handler\TransformHandler->execute()
#22 /srv/mediawiki/php-1.43.0-wmf.15/includes/Rest/Module/Module.php(269): MediaWiki\Rest\Module\Module->executeHandler(MWParsoid\Rest\Handler\TransformHandler)
#23 /srv/mediawiki/php-1.43.0-wmf.15/includes/Rest/Router.php(477): MediaWiki\Rest\Module\Module->execute(string, MediaWiki\Rest\RequestFromGlobals)
#24 /srv/mediawiki/php-1.43.0-wmf.15/includes/Rest/Router.php(446): MediaWiki\Rest\Router->doExecute(string, MediaWiki\Rest\RequestFromGlobals)
#25 /srv/mediawiki/php-1.43.0-wmf.15/includes/Rest/EntryPoint.php(209): MediaWiki\Rest\Router->execute(MediaWiki\Rest\RequestFromGlobals)
#26 /srv/mediawiki/php-1.43.0-wmf.15/includes/MediaWikiEntryPoint.php(200): MediaWiki\Rest\EntryPoint->execute()
#27 /srv/mediawiki/php-1.43.0-wmf.15/rest.php(39): MediaWiki\MediaWikiEntryPoint->run()
#28 /srv/mediawiki/w/rest.php(3): require(string)
#29 {main}
Impact
Notes

Maybe started around 1.43.0-wmf.4. 4591 of these since then.

Details

Request URL
https://ru.wikipedia.org/w/rest.php/ru.wikipedia.org/v3/transform/wikitext/to/pagebundle/Main_Page

Related Objects
Search...

Event Timeline

brennen created this task.Aug 1 2024, 3:18 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 1 2024, 3:18 PM
ABreault-WMF subscribed.Aug 1 2024, 10:11 PM

From T325761#8721649

The only utf-8 ones left now have always been for "Main Page" of ukwiki, ruwiki, bewiki. That indicates someone is POSTing some wikitext to the API which makes it harder to debug without having access to the wikitext in question. We have been ignoring this in the Parsoid logstash for a while now as possibly user error. But, maybe we should dump the wikitext being POSTed and see if we can figure out if it is an issue in Parsoid. But, need to figure out what the best way to do this is since the POSTed wikitext could be large blobs.

brennen moved this task from Backlog to Logs/Train on the User-brennen board.Aug 2 2024, 5:33 PM
jnuche triaged this task as Unbreak Now! priority.Aug 8 2024, 10:27 AM
jnuche mentioned this in T371875: Wikimedia\Assert\InvariantException: Invariant failed: Bad UTF-8 at start of string (parsoid/src/Html2Wt/SerializerState.php).
jnuche subscribed.

We got a continuous stream of this error and of T371875 after deploying wmf.17 to group2:

image.png (1,299×438 px, 68 KB)

Out of caution, I'm rolling back the train and setting these two errors as train blockers until someone can verify the actual impact

jnuche added a parent task: T366962: 1.43.0-wmf.17 deployment blockers.Aug 8 2024, 10:28 AM
jnuche lowered the priority of this task from Unbreak Now! to Needs Triage.Aug 8 2024, 12:29 PM

See convo in T371875

jnuche removed a parent task: T366962: 1.43.0-wmf.17 deployment blockers.Aug 8 2024, 12:29 PM
SLopes-WMF added a project: Content-Transform-Team-WIP.Aug 8 2024, 2:07 PM
SLopes-WMF moved this task from Backlog to Tracking on the Content-Platform-Team board.
SLopes-WMF moved this task from Needs Triage to Bugs & Crashers on the Parsoid board.Aug 8 2024, 2:17 PM
• thcipriani moved this task from Untriaged to Aug 2024 on the Wikimedia-production-error board.Aug 8 2024, 3:36 PM
MSantos added a project: Essential-Work.Aug 12 2024, 2:59 PM
Reedy subscribed.Mar 1 2025, 10:58 PM
Error
normalized_message
[{reqId}] {exception_url}   Wikimedia\Assert\InvariantException: Invariant failed: Bad UTF-8 at end of string (2 byte sequence)
FrameLocationCall
from/srv/mediawiki/php-1.44.0-wmf.18/vendor/wikimedia/assert/src/Assert.php(231)
#0/srv/mediawiki/php-1.44.0-wmf.18/vendor/wikimedia/parsoid/src/Utils/PHPUtils.php(204)Wikimedia\Assert\Assert::invariant(bool, string)
#1/srv/mediawiki/php-1.44.0-wmf.18/vendor/wikimedia/parsoid/src/Wt2Html/DOM/Processors/DOMRangeBuilder.php(954)Wikimedia\Parsoid\Utils\PHPUtils::safeSubstr(string, int, int)
#2/srv/mediawiki/php-1.44.0-wmf.18/vendor/wikimedia/parsoid/src/Wt2Html/DOM/Processors/DOMRangeBuilder.php(1273)Wikimedia\Parsoid\Wt2Html\DOM\Processors\DOMRangeBuilder->encapsulateTemplates(array)
#3/srv/mediawiki/php-1.44.0-wmf.18/vendor/wikimedia/parsoid/src/Wt2Html/DOM/Processors/WrapTemplates.php(26)Wikimedia\Parsoid\Wt2Html\DOM\Processors\DOMRangeBuilder->execute(Wikimedia\Parsoid\DOM\Element)
#4/srv/mediawiki/php-1.44.0-wmf.18/vendor/wikimedia/parsoid/src/Wt2Html/DOMProcessorPipeline.php(140)Wikimedia\Parsoid\Wt2Html\DOM\Processors\WrapTemplates->run(Wikimedia\Parsoid\Config\Env, Wikimedia\Parsoid\DOM\Element, array, bool)
#5/srv/mediawiki/php-1.44.0-wmf.18/vendor/wikimedia/parsoid/src/Wt2Html/DOMProcessorPipeline.php(171)Wikimedia\Parsoid\Wt2Html\DOMProcessorPipeline->doPostProcess(Wikimedia\Parsoid\DOM\Element)
#6/srv/mediawiki/php-1.44.0-wmf.18/vendor/wikimedia/parsoid/src/Wt2Html/DOMProcessorPipeline.php(189)Wikimedia\Parsoid\Wt2Html\DOMProcessorPipeline->process(Wikimedia\Parsoid\DOM\Element, array)
#7/srv/mediawiki/php-1.44.0-wmf.18/vendor/wikimedia/parsoid/src/Wt2Html/ParserPipeline.php(164)Wikimedia\Parsoid\Wt2Html\DOMProcessorPipeline->processChunkily(string, array)
#8/srv/mediawiki/php-1.44.0-wmf.18/vendor/wikimedia/parsoid/src/Wt2Html/ParserPipelineFactory.php(592)Wikimedia\Parsoid\Wt2Html\ParserPipeline->parseChunkily(string, array)
#9/srv/mediawiki/php-1.44.0-wmf.18/vendor/wikimedia/parsoid/src/Wikitext/ContentModelHandler.php(177)Wikimedia\Parsoid\Wt2Html\ParserPipelineFactory->parse(string)
#10/srv/mediawiki/php-1.44.0-wmf.18/vendor/wikimedia/parsoid/src/Parsoid.php(198)Wikimedia\Parsoid\Wikitext\ContentModelHandler->toDOM(Wikimedia\Parsoid\Ext\ParsoidExtensionAPI, null)
#11/srv/mediawiki/php-1.44.0-wmf.18/vendor/wikimedia/parsoid/src/Parsoid.php(263)Wikimedia\Parsoid\Parsoid->parseWikitext(MediaWiki\Parser\Parsoid\Config\PageConfig, MediaWiki\Parser\ParserOutput, array, null)
#12/srv/mediawiki/php-1.44.0-wmf.18/includes/parser/Parsoid/ParsoidParser.php(165)Wikimedia\Parsoid\Parsoid->wikitext2html(MediaWiki\Parser\Parsoid\Config\PageConfig, array, null, MediaWiki\Parser\ParserOutput)
#13/srv/mediawiki/php-1.44.0-wmf.18/includes/parser/Parsoid/ParsoidParser.php(294)MediaWiki\Parser\Parsoid\ParsoidParser->genParserOutput(MediaWiki\Parser\Parsoid\Config\PageConfig, MediaWiki\Parser\ParserOptions, null)
#14/srv/mediawiki/php-1.44.0-wmf.18/includes/content/WikitextContentHandler.php(384)MediaWiki\Parser\Parsoid\ParsoidParser->parse(string, MediaWiki\Title\Title, MediaWiki\Parser\ParserOptions, bool, bool, int, null)
#15/srv/mediawiki/php-1.44.0-wmf.18/includes/content/ContentHandler.php(1697)MediaWiki\Content\WikitextContentHandler->fillParserOutput(MediaWiki\Content\WikitextContent, MediaWiki\Content\Renderer\ContentParseParams, MediaWiki\Parser\ParserOutput)
#16/srv/mediawiki/php-1.44.0-wmf.18/includes/content/Renderer/ContentRenderer.php(75)MediaWiki\Content\ContentHandler->getParserOutput(MediaWiki\Content\WikitextContent, MediaWiki\Content\Renderer\ContentParseParams)
#17/srv/mediawiki/php-1.44.0-wmf.18/includes/Revision/RenderedRevision.php(261)MediaWiki\Content\Renderer\ContentRenderer->getParserOutput(MediaWiki\Content\WikitextContent, MediaWiki\Page\PageIdentityValue, MediaWiki\Revision\MutableRevisionRecord, MediaWiki\Parser\ParserOptions, array)
#18/srv/mediawiki/php-1.44.0-wmf.18/includes/Revision/RenderedRevision.php(233)MediaWiki\Revision\RenderedRevision->getSlotParserOutputUncached(MediaWiki\Content\WikitextContent, array)
#19/srv/mediawiki/php-1.44.0-wmf.18/includes/Revision/RevisionRenderer.php(236)MediaWiki\Revision\RenderedRevision->getSlotParserOutput(string, array)
#20/srv/mediawiki/php-1.44.0-wmf.18/includes/Revision/RevisionRenderer.php(169)MediaWiki\Revision\RevisionRenderer->combineSlotOutput(MediaWiki\Revision\RenderedRevision, MediaWiki\Parser\ParserOptions, array)
#21/srv/mediawiki/php-1.44.0-wmf.18/includes/Revision/RenderedRevision.php(196)MediaWiki\Revision\RevisionRenderer->MediaWiki\Revision\{closure}(MediaWiki\Revision\RenderedRevision, array)
#22/srv/mediawiki/php-1.44.0-wmf.18/includes/Rest/Handler/Helper/HtmlOutputRendererHelper.php(949)MediaWiki\Revision\RenderedRevision->getRevisionParserOutput()
#23/srv/mediawiki/php-1.44.0-wmf.18/includes/Rest/Handler/Helper/HtmlOutputRendererHelper.php(901)MediaWiki\Rest\Handler\Helper\HtmlOutputRendererHelper->parseUncacheable(MediaWiki\Page\PageStoreRecord, MediaWiki\Revision\MutableRevisionRecord, bool)
#24/srv/mediawiki/php-1.44.0-wmf.18/includes/Rest/Handler/Helper/HtmlOutputRendererHelper.php(641)MediaWiki\Rest\Handler\Helper\HtmlOutputRendererHelper->getParserOutputInternal()
#25/srv/mediawiki/php-1.44.0-wmf.18/includes/Rest/Handler/Helper/HtmlOutputRendererHelper.php(735)MediaWiki\Rest\Handler\Helper\HtmlOutputRendererHelper->getParserOutput()
#26/srv/mediawiki/php-1.44.0-wmf.18/includes/Rest/Handler/ParsoidHandler.php(739)MediaWiki\Rest\Handler\Helper\HtmlOutputRendererHelper->getPageBundle()
#27/srv/mediawiki/php-1.44.0-wmf.18/includes/Rest/Handler/TransformHandler.php(184)MediaWiki\Rest\Handler\ParsoidHandler->wt2html(MediaWiki\Parser\Parsoid\Config\PageConfig, array, string)
#28/srv/mediawiki/php-1.44.0-wmf.18/includes/Rest/Module/Module.php(415)MediaWiki\Rest\Handler\TransformHandler->execute()
#29/srv/mediawiki/php-1.44.0-wmf.18/includes/Rest/Module/Module.php(298)MediaWiki\Rest\Module\Module->executeHandler(MWParsoid\Rest\Handler\TransformHandler)
#30/srv/mediawiki/php-1.44.0-wmf.18/includes/Rest/Router.php(485)MediaWiki\Rest\Module\Module->execute(string, MediaWiki\Rest\RequestFromGlobals)
#31/srv/mediawiki/php-1.44.0-wmf.18/includes/Rest/Router.php(444)MediaWiki\Rest\Router->doExecute(string, MediaWiki\Rest\RequestFromGlobals)
#32/srv/mediawiki/php-1.44.0-wmf.18/includes/Rest/EntryPoint.php(209)MediaWiki\Rest\Router->execute(MediaWiki\Rest\RequestFromGlobals)
#33/srv/mediawiki/php-1.44.0-wmf.18/includes/MediaWikiEntryPoint.php(202)MediaWiki\Rest\EntryPoint->execute()
#34/srv/mediawiki/php-1.44.0-wmf.18/rest.php(39)MediaWiki\MediaWikiEntryPoint->run()
#35/srv/mediawiki/w/rest.php(3)require(string)
#36{main}
Impact
Notes
ihurbain removed a project: Content-Transform-Team-WIP.Apr 11 2025, 11:51 AM
OSleger-WMF subscribed.Jul 11 2025, 10:15 AM
OSleger-WMF added a comment.EditedJul 11 2025, 11:25 AM

Possible replication:

Template:Ta

<includeonly>{|
<div>
</includeonly>

Template:Tb

<includeonly>
</div>|}
</includeonly>

Page

{{Ta}}
řaa
<table>
<tr><td>
a
</td></tr>
</table>
{{Tb}}

Letter ř in this case can be replaced with any 2 byte utf-8 character.
Error stack trace:

Wikimedia\Assert\InvariantException from line 231 of /var/www/html/w/vendor/wikimedia/assert/src/Assert.php: Invariant failed: Bad UTF-8 at end of string (2 byte sequence)
#0 /Users/osleger/Projects/parsoid/src/Utils/PHPUtils.php(200): Wikimedia\Assert\Assert::invariant()
#1 /Users/osleger/Projects/parsoid/src/Wt2Html/DOM/Processors/DOMRangeBuilder.php(953): Wikimedia\Parsoid\Utils\PHPUtils::safeSubstr()
#2 /Users/osleger/Projects/parsoid/src/Wt2Html/DOM/Processors/DOMRangeBuilder.php(1278): Wikimedia\Parsoid\Wt2Html\DOM\Processors\DOMRangeBuilder->encapsulateTemplates()
#3 /Users/osleger/Projects/parsoid/src/Wt2Html/DOM/Processors/WrapTemplates.php(26): Wikimedia\Parsoid\Wt2Html\DOM\Processors\DOMRangeBuilder->execute()
#4 /Users/osleger/Projects/parsoid/src/Wt2Html/DOMProcessorPipeline.php(135): Wikimedia\Parsoid\Wt2Html\DOM\Processors\WrapTemplates->run()
#5 /Users/osleger/Projects/parsoid/src/Wt2Html/DOMProcessorPipeline.php(179): Wikimedia\Parsoid\Wt2Html\DOMProcessorPipeline->doPostProcess()
#6 /Users/osleger/Projects/parsoid/src/Wt2Html/DOMProcessorPipeline.php(200): Wikimedia\Parsoid\Wt2Html\DOMProcessorPipeline->process()
#7 /Users/osleger/Projects/parsoid/src/Wt2Html/ParserPipeline.php(165): Wikimedia\Parsoid\Wt2Html\DOMProcessorPipeline->processChunkily()
#8 /Users/osleger/Projects/parsoid/src/Wt2Html/ParserPipelineFactory.php(619): Wikimedia\Parsoid\Wt2Html\ParserPipeline->parseChunkily()
#9 /Users/osleger/Projects/parsoid/src/Wikitext/ContentModelHandler.php(185): Wikimedia\Parsoid\Wt2Html\ParserPipelineFactory->parse()
#10 /Users/osleger/Projects/parsoid/src/Parsoid.php(198): Wikimedia\Parsoid\Wikitext\ContentModelHandler->toDOM()
#11 /Users/osleger/Projects/parsoid/src/Parsoid.php(264): Wikimedia\Parsoid\Parsoid->parseWikitext()
#12 /var/www/html/w/includes/parser/Parsoid/ParsoidParser.php(165): Wikimedia\Parsoid\Parsoid->wikitext2html()
#13 /var/www/html/w/includes/parser/Parsoid/ParsoidParser.php(294): MediaWiki\Parser\Parsoid\ParsoidParser->genParserOutput()
#14 /var/www/html/w/maintenance/parse.php(97): MediaWiki\Parser\Parsoid\ParsoidParser->parse()
#15 /var/www/html/w/maintenance/parse.php(87): CLIParser->render()
#16 /var/www/html/w/maintenance/includes/MaintenanceRunner.php(691): CLIParser->execute()
#17 /var/www/html/w/maintenance/run.php(51): MediaWiki\Maintenance\MaintenanceRunner->run()
#18 {main}

Link to production error:
https://outreach.wikimedia.org/w/index.php?oldid=45210&useparsoid=1

OSleger-WMF claimed this task.Jul 11 2025, 8:29 PM
jnuche merged a task: T389688: Wikimedia\Assert\InvariantException: Invariant failed: Bad UTF-8 at end of string (3 byte sequence).Oct 9 2025, 11:11 AM
jnuche added a subscriber: ssastry.
jnuche mentioned this in T406852: Wikimedia\Assert\InvariantException: Invariant failed: Bad UTF-8 at start of string.Oct 9 2025, 11:45 AM
ihurbain mentioned this in T410286: 500 internal service error.Nov 17 2025, 4:48 PM
gerritbot added a comment.Nov 18 2025, 3:31 PM

Change #1206896 had a related patch set uploaded (by C. Scott Ananian; author: C. Scott Ananian):

[mediawiki/services/parsoid@master] WIP: Reproduce T371617

https://gerrit.wikimedia.org/r/1206896

gerritbot added a project: Patch-For-Review.Nov 18 2025, 3:31 PM
OSleger-WMF added a comment.Nov 18 2025, 3:41 PM

I was debuging this scenario, figured out what is wrong, but I am not sure, what parsoid should in this scenario do. Problem is like this:

Into the tree builder, this set of tokens comes:

0-[HTML]       | {"type":"SelfclosingTagTk","name":"meta","attribs":[{"k":"typeof","v":"mw:Transclusion"},{"k":"about","v":"#mwt1"}],"dataParsoid":{"tsr":[0,6],"src":"{{Ta}}"}}
0-[HTML]       | {"type":"TagTk","name":"table","attribs":[],"dataParsoid":{}}
0-[HTML]       | {"type":"NlTk","dataParsoid":{}}
0-[HTML]       | {"type":"TagTk","name":"div","attribs":[],"dataParsoid":{"stx":"html"}}
0-[HTML]       | {"type":"NlTk","dataParsoid":{}}
0-[HTML]       | {"type":"SelfclosingTagTk","name":"meta","attribs":[{"k":"typeof","v":"mw:Transclusion/End"},{"k":"about","v":"#mwt1"}],"dataParsoid":{"tsr":[null,6]}}
0-[HTML]       | {"type":"NlTk","dataParsoid":{"tsr":[6,7]}}
0-[HTML]       | {"type":"TagTk","name":"p","attribs":[],"dataParsoid":{}}
0-[HTML]       | řaaaaaaa
0-[HTML]       | {"type":"EndTagTk","name":"p","attribs":[],"dataParsoid":{}}

Everything like I would expect, but in tree builder, this DOM is generated:

<meta typeof="mw:Transclusion" about="#mwt1" data-parsoid='{"tsr":[0,6],"src":"{{Ta}}"}'/><div data-parsoid='{"stx":"html","autoInsertedEnd":true}'>
<meta typeof="mw:Transclusion/End" about="#mwt1" data-parsoid='{"tsr":[null,6]}'/>
<p data-parsoid="{}">řaaaaaaa</p>

a</div><table data-parsoid='{"autoInsertedEnd":true}'>
</table>

At first I thought, that moved table node under the paragraph is problem, but when compared to behavior of legacy parser, it is expected. What is not expected is that table is removed from template meta tags, by which is table recognized as part of template.

Then when DSR calculation is in process, table takes bytes from original text, like it was not part of the template. It takes 2 bytes for table macro ("{|") and 1 byte for newline present inside of table (in template).

Then template wrapper comes in, and uses wrongly calculated DSR to split text and if luck is not on Parsoid side, it will split UTF-8 character and goes to exception.

What I am not sure is, how Should result look like

JArguello-WMF subscribed.Nov 18 2025, 10:28 PM
cscott edited projects, added: Content-Platform-Team (Work In Progress); removed: Content-Platform-Team.Dec 1 2025, 4:29 PM
JArguello-WMF added a comment.Dec 8 2025, 2:46 PM

hi team! just a heads up that there are more errors and we updated the ticket https://phabricator.wikimedia.org/T410286
Thanks for your help!

JArguello-WMF added a comment.Dec 10 2025, 3:00 PM

Hi Everyone,
We are still getting the above errors.

"wmf api call returned status not 200, 206 or 302 for url https://fr.wikipedia.org/w/rest.php/v1/revision/231309717/html? with error 500 Internal Server Error:{\"message\":\"Error:exceptionoftypeWikimedia\\\\Assert\\\\InvariantException\",\"reqId\":\"c2bb862e-608d-4aff-8e81-ba0271efede6\",\"httpCode\":500,\"httpReason\":\"InternalServerError\"}"

your help is very appreciated. Thanks!

RThomas-WMF subscribed.Jan 16 2026, 1:06 PM

Another instance of the error

https://fr.wikipedia.org/w/rest.php/v1/revision/232458284/html? with error 500 Internal Server Error:{\"message\":\"Error:exceptionoftypeWikimedia\\\\Assert\\\\InvariantException\",\"reqId\":\"ce9043d1-b6b8-4bb1-8795-5850f722e6be\",\"httpCode\":500,\"httpReason\":\"InternalServerError\"}",
cscott moved this task from Backlog to In Progress on the Content-Platform-Team (Work In Progress) board.Mar 9 2026, 3:17 PM
OSleger-WMF added a parent task: T421693: Bad UTF-8 errors.Mar 30 2026, 11:26 AM
RThomas-WMF added a comment.May 15 2026, 1:19 PM
url https://ba.wikipedia.org/w/rest.php/v1/revision/1296939/html? with error 500 Internal Server Error:{\"message\":\"Error:exceptionoftypeWikimedia\\\\Assert\\\\InvariantException\",\"reqId\":\"66e89395-19ef-4c7e-9b2d-70c3ad1e2495\",\"httpCode\":500,\"httpReason\":\"InternalServerError\"}",
JArguello-WMF added a project: Wikimedia Enterprise.May 15 2026, 1:21 PM
JArguello-WMF moved this task from Incoming to Radar on the Wikimedia Enterprise board.
Zabe mentioned this in T430727: Internal error in ruwiki.Jun 30 2026, 9:31 PM
• thcipriani merged a task: T430727: Internal error in ruwiki.Jul 2 2026, 3:22 PM
• thcipriani added subscribers: Wellverywell, Zabe, Futbollo.
• thcipriani subscribed.

Fresh stack trace from ruwiki on T430727:

==== Error ====

* mwversion: 1.47.0-wmf.8
* timestamp: 2026-06-30T18:26:39.588Z
* phpversion: `8.3.31`
* reqId: `c729eae1-37e5-48db-af72-a4711036788c`
* [[ https://logstash.wikimedia.org/app/dashboards#/view/AXFV7JE83bOlOASGccsT?_g=(time:(from:'2026-06-29T18:26:39.588Z',to:'2026-07-01T18:26:39.588Z'))&_a=(query:(query_string:(query:'reqId:%22c729eae1-37e5-48db-af72-a4711036788c%22'))) | Find reqId in Logstash ]]

```name=normalized_message,lines=10
[{reqId}] {exception_url}   Wikimedia\Assert\InvariantException: Invariant failed: Bad UTF-8 at end of string (2 byte sequence)
FrameLocationCall
from/srv/mediawiki/php-1.47.0-wmf.8/vendor/wikimedia/assert/src/Assert.php(231)
#0/srv/mediawiki/php-1.47.0-wmf.8/vendor/wikimedia/parsoid/src/Utils/PHPUtils.php(182)Wikimedia\Assert\Assert::invariant(bool, string)
#1/srv/mediawiki/php-1.47.0-wmf.8/vendor/wikimedia/parsoid/src/Core/SourceRange.php(104)Wikimedia\Parsoid\Utils\PHPUtils::safeSubstr(string, int, int)
#2/srv/mediawiki/php-1.47.0-wmf.8/vendor/wikimedia/parsoid/src/Wt2Html/TT/TemplateEncapsulator.php(156)Wikimedia\Parsoid\Core\SourceRange->substr(Wikimedia\Parsoid\Core\SourceString)
#3/srv/mediawiki/php-1.47.0-wmf.8/vendor/wikimedia/parsoid/src/Wt2Html/TT/TemplateEncapsulator.php(77)Wikimedia\Parsoid\Wt2Html\TT\TemplateEncapsulator->getTemplateInfo()
#4/srv/mediawiki/php-1.47.0-wmf.8/vendor/wikimedia/parsoid/src/Wt2Html/TT/TemplateHandler.php(692)Wikimedia\Parsoid\Wt2Html\TT\TemplateEncapsulator->encapTokens(array)
#5/srv/mediawiki/php-1.47.0-wmf.8/vendor/wikimedia/parsoid/src/Wt2Html/TT/TemplateHandler.php(1119)Wikimedia\Parsoid\Wt2Html\TT\TemplateHandler->encapTokens(Wikimedia\Parsoid\Wt2Html\TT\TemplateEncapsulator, array)
#6/srv/mediawiki/php-1.47.0-wmf.8/vendor/wikimedia/parsoid/src/Wt2Html/TT/TemplateHandler.php(1156)Wikimedia\Parsoid\Wt2Html\TT\TemplateHandler->onTemplate(Wikimedia\Parsoid\Tokens\SelfclosingTagTk)
#7/srv/mediawiki/php-1.47.0-wmf.8/vendor/wikimedia/parsoid/src/Wt2Html/TT/XMLTagBasedHandler.php(47)Wikimedia\Parsoid\Wt2Html\TT\TemplateHandler->onTag(Wikimedia\Parsoid\Tokens\SelfclosingTagTk)
#8/srv/mediawiki/php-1.47.0-wmf.8/vendor/wikimedia/parsoid/src/Wt2Html/TokenHandlerPipeline.php(116)Wikimedia\Parsoid\Wt2Html\TT\XMLTagBasedHandler->process(array)
#9/srv/mediawiki/php-1.47.0-wmf.8/vendor/wikimedia/parsoid/src/Wt2Html/TokenHandlerPipeline.php(170)Wikimedia\Parsoid\Wt2Html\TokenHandlerPipeline->processChunk(array)
#10/srv/mediawiki/php-1.47.0-wmf.8/vendor/wikimedia/parsoid/src/Wt2Html/ParserPipeline.php(127)Wikimedia\Parsoid\Wt2Html\TokenHandlerPipeline->process(array, array)
#11/srv/mediawiki/php-1.47.0-wmf.8/vendor/wikimedia/parsoid/src/Utils/PipelineUtils.php(139)Wikimedia\Parsoid\Wt2Html\ParserPipeline->parse(string, array)
#12/srv/mediawiki/php-1.47.0-wmf.8/vendor/wikimedia/parsoid/src/Wt2Html/DOM/Handlers/TableFixups.php(527)Wikimedia\Parsoid\Utils\PipelineUtils::processContentInPipeline(Wikimedia\Parsoid\Config\Env, Wikimedia\Parsoid\Wt2Html\PageConfigFrame, string, array)
#13/srv/mediawiki/php-1.47.0-wmf.8/vendor/wikimedia/parsoid/src/Wt2Html/DOM/Handlers/TableFixups.php(903)Wikimedia\Parsoid\Wt2Html\DOM\Handlers\TableFixups::convertAttribsToContent(Wikimedia\Parsoid\Config\Env, Wikimedia\Parsoid\Wt2Html\PageConfigFrame, Wikimedia\Parsoid\DOM\Element, bool, bool)
#14/srv/mediawiki/php-1.47.0-wmf.8/vendor/wikimedia/parsoid/src/Utils/DOMTraverser.php(143)Wikimedia\Parsoid\Wt2Html\DOM\Handlers\TableFixups::handleTableCellTemplates(Wikimedia\Parsoid\DOM\Element, Wikimedia\Parsoid\Utils\DTState)
#15/srv/mediawiki/php-1.47.0-wmf.8/vendor/wikimedia/parsoid/src/Utils/DOMTraverser.php(221)Wikimedia\Parsoid\Utils\DOMTraverser->callHandlers(Wikimedia\Parsoid\DOM\Element, MediaWiki\Parser\Parsoid\Config\SiteConfig, Wikimedia\Parsoid\Utils\DTState)
#16/srv/mediawiki/php-1.47.0-wmf.8/vendor/wikimedia/parsoid/src/Utils/DOMTraverser.php(235)Wikimedia\Parsoid\Utils\DOMTraverser->traverseInternal(bool, MediaWiki\Parser\Parsoid\Config\SiteConfig, Wikimedia\Parsoid\DOM\Element, Wikimedia\Parsoid\Utils\DTState)
#17/srv/mediawiki/php-1.47.0-wmf.8/vendor/wikimedia/parsoid/src/Utils/DOMTraverser.php(235)Wikimedia\Parsoid\Utils\DOMTraverser->traverseInternal(bool, MediaWiki\Parser\Parsoid\Config\SiteConfig, Wikimedia\Parsoid\DOM\Element, Wikimedia\Parsoid\Utils\DTState)
#18/srv/mediawiki/php-1.47.0-wmf.8/vendor/wikimedia/parsoid/src/Utils/DOMTraverser.php(235)Wikimedia\Parsoid\Utils\DOMTraverser->traverseInternal(bool, MediaWiki\Parser\Parsoid\Config\SiteConfig, Wikimedia\Parsoid\DOM\Element, Wikimedia\Parsoid\Utils\DTState)
#19/srv/mediawiki/php-1.47.0-wmf.8/vendor/wikimedia/parsoid/src/Utils/DOMTraverser.php(235)Wikimedia\Parsoid\Utils\DOMTraverser->traverseInternal(bool, MediaWiki\Parser\Parsoid\Config\SiteConfig, Wikimedia\Parsoid\DOM\Element, Wikimedia\Parsoid\Utils\DTState)
#20/srv/mediawiki/php-1.47.0-wmf.8/vendor/wikimedia/parsoid/src/Utils/DOMTraverser.php(177)Wikimedia\Parsoid\Utils\DOMTraverser->traverseInternal(bool, MediaWiki\Parser\Parsoid\Config\SiteConfig, Wikimedia\Parsoid\DOM\Element, Wikimedia\Parsoid\Utils\DTState)
#21/srv/mediawiki/php-1.47.0-wmf.8/vendor/wikimedia/parsoid/src/Wt2Html/DOM/Processors/DOMPPTraverser.php(42)Wikimedia\Parsoid\Utils\DOMTraverser->traverse(MediaWiki\Parser\Parsoid\Config\SiteConfig, Wikimedia\Parsoid\DOM\Element, Wikimedia\Parsoid\Utils\DTState)
#22/srv/mediawiki/php-1.47.0-wmf.8/vendor/wikimedia/parsoid/src/Wt2Html/DOMProcessorPipeline.php(131)Wikimedia\Parsoid\Wt2Html\DOM\Processors\DOMPPTraverser->run(Wikimedia\Parsoid\Config\Env, Wikimedia\Parsoid\DOM\Element, array, bool)
#23/srv/mediawiki/php-1.47.0-wmf.8/vendor/wikimedia/parsoid/src/Wt2Html/DOMProcessorPipeline.php(175)Wikimedia\Parsoid\Wt2Html\DOMProcessorPipeline->doPostProcess(Wikimedia\Parsoid\DOM\Element)
#24/srv/mediawiki/php-1.47.0-wmf.8/vendor/wikimedia/parsoid/src/Wt2Html/DOMProcessorPipeline.php(188)Wikimedia\Parsoid\Wt2Html\DOMProcessorPipeline->process(Wikimedia\Parsoid\DOM\Element, array)
#25/srv/mediawiki/php-1.47.0-wmf.8/vendor/wikimedia/parsoid/src/Wt2Html/ParserPipeline.php(159)Wikimedia\Parsoid\Wt2Html\DOMProcessorPipeline->processChunkily(Wikimedia\Parsoid\DOM\Element, array)
#26/srv/mediawiki/php-1.47.0-wmf.8/vendor/wikimedia/parsoid/src/Wt2Html/ParserPipeline.php(181)Wikimedia\Parsoid\Wt2Html\ParserPipeline->processPipelineStages(array, string, array)
#27/srv/mediawiki/php-1.47.0-wmf.8/vendor/wikimedia/parsoid/src/Wt2Html/ParserPipelineFactory.php(622)Wikimedia\Parsoid\Wt2Html\ParserPipeline->parseChunkily(string, array)
#28/srv/mediawiki/php-1.47.0-wmf.8/vendor/wikimedia/parsoid/src/Wikitext/ContentModelHandler.php(192)Wikimedia\Parsoid\Wt2Html\ParserPipelineFactory->parse(string)
#29/srv/mediawiki/php-1.47.0-wmf.8/vendor/wikimedia/parsoid/src/Parsoid.php(204)Wikimedia\Parsoid\Wikitext\ContentModelHandler->toDOM(Wikimedia\Parsoid\Ext\ParsoidExtensionAPI, null)
#30/srv/mediawiki/php-1.47.0-wmf.8/vendor/wikimedia/parsoid/src/Parsoid.php(270)Wikimedia\Parsoid\Parsoid->parseWikitext(MediaWiki\Parser\Parsoid\Config\PageConfig, MediaWiki\Parser\ParserOutput, array, null)
#31/srv/mediawiki/php-1.47.0-wmf.8/includes/Parser/Parsoid/ParsoidParser.php(155)Wikimedia\Parsoid\Parsoid->wikitext2html(MediaWiki\Parser\Parsoid\Config\PageConfig, array, null, MediaWiki\Parser\ParserOutput)
#32/srv/mediawiki/php-1.47.0-wmf.8/includes/Parser/Parsoid/ParsoidParser.php(321)MediaWiki\Parser\Parsoid\ParsoidParser->genParserOutput(MediaWiki\Parser\Parsoid\Config\PageConfig, MediaWiki\Parser\ParserOptions, null)
#33/srv/mediawiki/php-1.47.0-wmf.8/includes/Content/WikitextContentHandler.php(360)MediaWiki\Parser\Parsoid\ParsoidParser->parse(string, MediaWiki\Title\Title, MediaWiki\Parser\ParserOptions, bool, bool, int, null)
#34/srv/mediawiki/php-1.47.0-wmf.8/includes/Content/ContentHandler.php(1574)MediaWiki\Content\WikitextContentHandler->fillParserOutput(MediaWiki\Content\WikitextContent, MediaWiki\Content\Renderer\ContentParseParams, MediaWiki\Parser\ParserOutput)
#35/srv/mediawiki/php-1.47.0-wmf.8/includes/Content/Renderer/ContentRenderer.php(62)MediaWiki\Content\ContentHandler->getParserOutput(MediaWiki\Content\WikitextContent, MediaWiki\Content\Renderer\ContentParseParams)
#36/srv/mediawiki/php-1.47.0-wmf.8/extensions/FlaggedRevs/includes/backend/FlaggedRevs.php(369)MediaWiki\Content\Renderer\ContentRenderer->getParserOutput(MediaWiki\Content\WikitextContent, MediaWiki\Title\Title, MediaWiki\Revision\RevisionStoreRecord, MediaWiki\Parser\ParserOptions)
#37/srv/mediawiki/php-1.47.0-wmf.8/extensions/FlaggedRevs/includes/backend/FlaggedRevs.php(279)FlaggedRevs::parseStableRevision(FlaggedRevision, MediaWiki\Parser\ParserOptions)
#38/srv/mediawiki/php-1.47.0-wmf.8/includes/PoolCounter/PoolCounterWorkViaCallback.php(68)FlaggedRevs::{closure}()
#39/srv/mediawiki/php-1.47.0-wmf.8/includes/PoolCounter/PoolCounterWork.php(159)MediaWiki\PoolCounter\PoolCounterWorkViaCallback->doWork()
#40/srv/mediawiki/php-1.47.0-wmf.8/extensions/FlaggedRevs/includes/backend/FlaggedRevs.php(298)MediaWiki\PoolCounter\PoolCounterWork->execute()
#41/srv/mediawiki/php-1.47.0-wmf.8/extensions/FlaggedRevs/includes/frontend/FlaggablePageView.php(656)FlaggedRevs::parseStableRevisionPooled(FlaggedRevision, MediaWiki\Parser\ParserOptions)
#42/srv/mediawiki/php-1.47.0-wmf.8/extensions/FlaggedRevs/includes/frontend/FlaggablePageView.php(339)FlaggablePageView->showStableVersion(FlaggedRevision, string)
#43/srv/mediawiki/php-1.47.0-wmf.8/extensions/FlaggedRevs/includes/frontend/FlaggedRevsUIHooks.php(288)FlaggablePageView->setPageContent(null, bool)
#44/srv/mediawiki/php-1.47.0-wmf.8/includes/HookContainer/HookContainer.php(127)FlaggedRevsUIHooks->onArticleViewHeader(MediaWiki\Page\Article, null, bool)
#45/srv/mediawiki/php-1.47.0-wmf.8/includes/HookContainer/HookRunner.php(897)MediaWiki\HookContainer\HookContainer->run(string, array)
#46/srv/mediawiki/php-1.47.0-wmf.8/includes/Page/Article.php(706)MediaWiki\HookContainer\HookRunner->onArticleViewHeader(MediaWiki\Page\Article, null, bool)
#47/srv/mediawiki/php-1.47.0-wmf.8/includes/Page/Article.php(546)MediaWiki\Page\Article->generateContentOutput(MediaWiki\User\User, MediaWiki\Parser\ParserOptions, int, MediaWiki\Output\OutputPage, array)
#48/srv/mediawiki/php-1.47.0-wmf.8/includes/Actions/ViewAction.php(71)MediaWiki\Page\Article->view()
#49/srv/mediawiki/php-1.47.0-wmf.8/includes/Actions/ActionEntryPoint.php(753)MediaWiki\Actions\ViewAction->show()
#50/srv/mediawiki/php-1.47.0-wmf.8/includes/Actions/ActionEntryPoint.php(511)MediaWiki\Actions\ActionEntryPoint->performAction(MediaWiki\Page\Article, MediaWiki\Title\Title)
#51/srv/mediawiki/php-1.47.0-wmf.8/includes/Actions/ActionEntryPoint.php(145)MediaWiki\Actions\ActionEntryPoint->performRequest()
#52/srv/mediawiki/php-1.47.0-wmf.8/includes/MediaWikiEntryPoint.php(180)MediaWiki\Actions\ActionEntryPoint->execute()
#53/srv/mediawiki/php-1.47.0-wmf.8/index.php(44)MediaWiki\MediaWikiEntryPoint->run()
#54/srv/mediawiki/w/index.php(3)require(string)
#55{main}
ssastry added a comment.Jul 2 2026, 6:36 PM


In T371617#12081565, @thcipriani wrote:


Fresh stack trace from ruwiki on T430727:





This is fixed as part of T430186: Wikimedia\Assert\InvariantException: Invariant failed: Bad UTF-8 at start of string
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits