Maniphest T301948

Wikimedia\Assert\InvariantException: Invariant failed: No KVs expected.
Closed, ResolvedPublicPRODUCTION ERROR
Actions

Assigned To

None

Authored By

	hashar
	Feb 17 2022, 8:43 AM

Description

Error

mwversion: 1.38.0-wmf.22
reqId: 8162dbd4-b1f8-4742-a6a3-90fdd331db11
Find reqId in Logstash

normalized_message

[{reqId}] {exception_url}   Wikimedia\Assert\InvariantException: Invariant failed: No KVs expected.

exception.trace

from /srv/mediawiki/php-1.38.0-wmf.22/vendor/wikimedia/assert/src/Assert.php(231)
#0 /srv/mediawiki/php-1.38.0-wmf.22/vendor/wikimedia/parsoid/src/Utils/PHPUtils.php(390): Wikimedia\Assert\Assert::invariant(boolean, string)
#1 /srv/mediawiki/php-1.38.0-wmf.22/vendor/wikimedia/parsoid/src/Utils/TokenUtils.php(611): Wikimedia\Parsoid\Utils\PHPUtils::unreachable(string)
#2 /srv/mediawiki/php-1.38.0-wmf.22/vendor/wikimedia/parsoid/src/Wt2Html/TT/SanitizerHandler.php(74): Wikimedia\Parsoid\Utils\TokenUtils::tokensToString(array)
#3 /srv/mediawiki/php-1.38.0-wmf.22/vendor/wikimedia/parsoid/src/Wt2Html/TT/SanitizerHandler.php(148): Wikimedia\Parsoid\Wt2Html\TT\SanitizerHandler->sanitizeToken(MWParsoid\Config\SiteConfig, Wikimedia\Parsoid\Wt2Html\Frame, Wikimedia\Parsoid\Tokens\SelfclosingTagTk, boolean)
#4 /srv/mediawiki/php-1.38.0-wmf.22/vendor/wikimedia/parsoid/src/Wt2Html/TT/TokenHandler.php(159): Wikimedia\Parsoid\Wt2Html\TT\SanitizerHandler->onAny(Wikimedia\Parsoid\Tokens\SelfclosingTagTk)
#5 /srv/mediawiki/php-1.38.0-wmf.22/vendor/wikimedia/parsoid/src/Wt2Html/TokenTransformManager.php(132): Wikimedia\Parsoid\Wt2Html\TT\TokenHandler->process(array)
#6 /srv/mediawiki/php-1.38.0-wmf.22/vendor/wikimedia/parsoid/src/Wt2Html/TokenTransformManager.php(185): Wikimedia\Parsoid\Wt2Html\TokenTransformManager->processChunk(array)
#7 /srv/mediawiki/php-1.38.0-wmf.22/vendor/wikimedia/parsoid/src/Wt2Html/ParserPipeline.php(140): Wikimedia\Parsoid\Wt2Html\TokenTransformManager->process(array, array)
#8 /srv/mediawiki/php-1.38.0-wmf.22/vendor/wikimedia/parsoid/src/Utils/PipelineUtils.php(105): Wikimedia\Parsoid\Wt2Html\ParserPipeline->parse(string, array)
#9 /srv/mediawiki/php-1.38.0-wmf.22/vendor/wikimedia/parsoid/src/Ext/ParsoidExtensionAPI.php(288): Wikimedia\Parsoid\Utils\PipelineUtils::processContentInPipeline(Wikimedia\Parsoid\Config\Env, Wikimedia\Parsoid\Wt2Html\Frame, string, array)
#10 /srv/mediawiki/php-1.38.0-wmf.22/vendor/wikimedia/parsoid/src/Ext/ParsoidExtensionAPI.php(336): Wikimedia\Parsoid\Ext\ParsoidExtensionAPI->wikitextToDOM(string, array, boolean)
#11 /srv/mediawiki/php-1.38.0-wmf.22/vendor/wikimedia/parsoid/src/Ext/Cite/References.php(712): Wikimedia\Parsoid\Ext\ParsoidExtensionAPI->extTagToDOM(array, string, string, array)
#12 /srv/mediawiki/php-1.38.0-wmf.22/vendor/wikimedia/parsoid/src/Wt2Html/TT/ExtensionHandler.php(106): Wikimedia\Parsoid\Ext\Cite\References->sourceToDom(Wikimedia\Parsoid\Ext\ParsoidExtensionAPI, string, array)
#13 /srv/mediawiki/php-1.38.0-wmf.22/vendor/wikimedia/parsoid/src/Wt2Html/TT/ExtensionHandler.php(290): Wikimedia\Parsoid\Wt2Html\TT\ExtensionHandler->onExtension(Wikimedia\Parsoid\Tokens\SelfclosingTagTk)
#14 /srv/mediawiki/php-1.38.0-wmf.22/vendor/wikimedia/parsoid/src/Wt2Html/TT/TokenHandler.php(150): Wikimedia\Parsoid\Wt2Html\TT\ExtensionHandler->onTag(Wikimedia\Parsoid\Tokens\SelfclosingTagTk)
#15 /srv/mediawiki/php-1.38.0-wmf.22/vendor/wikimedia/parsoid/src/Wt2Html/TokenTransformManager.php(132): Wikimedia\Parsoid\Wt2Html\TT\TokenHandler->process(array)
#16 /srv/mediawiki/php-1.38.0-wmf.22/vendor/wikimedia/parsoid/src/Wt2Html/TokenTransformManager.php(185): Wikimedia\Parsoid\Wt2Html\TokenTransformManager->processChunk(array)
#17 /srv/mediawiki/php-1.38.0-wmf.22/vendor/wikimedia/parsoid/src/Wt2Html/ParserPipeline.php(140): Wikimedia\Parsoid\Wt2Html\TokenTransformManager->process(array, array)
#18 /srv/mediawiki/php-1.38.0-wmf.22/vendor/wikimedia/parsoid/src/Utils/PipelineUtils.php(105): Wikimedia\Parsoid\Wt2Html\ParserPipeline->parse(string, array)
#19 /srv/mediawiki/php-1.38.0-wmf.22/vendor/wikimedia/parsoid/src/Wt2Html/TT/TemplateHandler.php(553): Wikimedia\Parsoid\Utils\PipelineUtils::processContentInPipeline(Wikimedia\Parsoid\Config\Env, Wikimedia\Parsoid\Wt2Html\PageConfigFrame, string, array)
#20 /srv/mediawiki/php-1.38.0-wmf.22/vendor/wikimedia/parsoid/src/Wt2Html/TT/TemplateHandler.php(1105): Wikimedia\Parsoid\Wt2Html\TT\TemplateHandler->processTemplateSource(array, array, string)
#21 /srv/mediawiki/php-1.38.0-wmf.22/vendor/wikimedia/parsoid/src/Wt2Html/TT/TemplateHandler.php(1160): Wikimedia\Parsoid\Wt2Html\TT\TemplateHandler->onTemplate(Wikimedia\Parsoid\Tokens\SelfclosingTagTk)
#22 /srv/mediawiki/php-1.38.0-wmf.22/vendor/wikimedia/parsoid/src/Wt2Html/TT/TokenHandler.php(150): Wikimedia\Parsoid\Wt2Html\TT\TemplateHandler->onTag(Wikimedia\Parsoid\Tokens\SelfclosingTagTk)
#23 /srv/mediawiki/php-1.38.0-wmf.22/vendor/wikimedia/parsoid/src/Wt2Html/TokenTransformManager.php(132): Wikimedia\Parsoid\Wt2Html\TT\TokenHandler->process(array)
#24 /srv/mediawiki/php-1.38.0-wmf.22/vendor/wikimedia/parsoid/src/Wt2Html/TokenTransformManager.php(195): Wikimedia\Parsoid\Wt2Html\TokenTransformManager->processChunk(array)
#25 /srv/mediawiki/php-1.38.0-wmf.22/vendor/wikimedia/parsoid/src/Wt2Html/TokenTransformManager.php(193): Wikimedia\Parsoid\Wt2Html\TokenTransformManager->processChunkily(string, array)
#26 /srv/mediawiki/php-1.38.0-wmf.22/vendor/wikimedia/parsoid/src/Wt2Html/TreeBuilder/TreeBuilderStage.php(487): Wikimedia\Parsoid\Wt2Html\TokenTransformManager->processChunkily(string, array)
#27 [internal function]: Wikimedia\Parsoid\Wt2Html\TreeBuilder\TreeBuilderStage->processChunkily(string, array)
#28 /srv/mediawiki/php-1.38.0-wmf.22/vendor/wikimedia/parsoid/src/Wt2Html/DOMPostProcessor.php(1040): Generator->current()
#29 /srv/mediawiki/php-1.38.0-wmf.22/vendor/wikimedia/parsoid/src/Wt2Html/ParserPipeline.php(180): Wikimedia\Parsoid\Wt2Html\DOMPostProcessor->processChunkily(string, array)
#30 /srv/mediawiki/php-1.38.0-wmf.22/vendor/wikimedia/parsoid/src/Wt2Html/ParserPipelineFactory.php(308): Wikimedia\Parsoid\Wt2Html\ParserPipeline->parseChunkily(string, array)
#31 /srv/mediawiki/php-1.38.0-wmf.22/vendor/wikimedia/parsoid/src/Wikitext/ContentModelHandler.php(122): Wikimedia\Parsoid\Wt2Html\ParserPipelineFactory->parse(string)
#32 /srv/mediawiki/php-1.38.0-wmf.22/vendor/wikimedia/parsoid/src/Parsoid.php(168): Wikimedia\Parsoid\Wikitext\ContentModelHandler->toDOM(Wikimedia\Parsoid\Ext\ParsoidExtensionAPI)
#33 /srv/mediawiki/php-1.38.0-wmf.22/vendor/wikimedia/parsoid/src/Parsoid.php(200): Wikimedia\Parsoid\Parsoid->parseWikitext(MWParsoid\Config\PageConfig, array)
#34 /srv/mediawiki/php-1.38.0-wmf.22/vendor/wikimedia/parsoid/extension/src/Rest/Handler/ParsoidHandler.php(584): Wikimedia\Parsoid\Parsoid->wikitext2html(MWParsoid\Config\PageConfig, array, NULL)
#35 /srv/mediawiki/php-1.38.0-wmf.22/vendor/wikimedia/parsoid/extension/src/Rest/Handler/PageHandler.php(88): MWParsoid\Rest\Handler\ParsoidHandler->wt2html(MWParsoid\Config\PageConfig, array)
#36 /srv/mediawiki/php-1.38.0-wmf.22/includes/Rest/Router.php(414): MWParsoid\Rest\Handler\PageHandler->execute()
#37 /srv/mediawiki/php-1.38.0-wmf.22/includes/Rest/Router.php(338): MediaWiki\Rest\Router->executeHandler(MWParsoid\Rest\Handler\PageHandler)
#38 /srv/mediawiki/php-1.38.0-wmf.22/includes/Rest/EntryPoint.php(167): MediaWiki\Rest\Router->execute(MediaWiki\Rest\RequestFromGlobals)
#39 /srv/mediawiki/php-1.38.0-wmf.22/includes/Rest/EntryPoint.php(132): MediaWiki\Rest\EntryPoint->execute()
#40 /srv/mediawiki/php-1.38.0-wmf.22/rest.php(31): MediaWiki\Rest\EntryPoint::main()
#41 /srv/mediawiki/w/rest.php(3): require(string)
#42 {main}

Impact

Probably not a big deal since all messages apparently came from a single request (reqid 8162dbd4-b1f8-4742-a6a3-90fdd331db11).

It is none the less a condition that should be handled?

Notes

We had ~ 5500 such events all in a short burst around Feb 17, 2022 @ 00:29. The wiki was lawiktionary

Details

Request URL: https://la.wiktionary.org/w/rest.php/la.wiktionary.org/v3/page/pagebundle/agricola/212180

Related Changes in Gerrit:

Subject	Repo	Branch	Lines +/-
Remove shuttling of template argument tokens to the end of the stage	mediawiki/services/parsoid	master	+5 -3
Bump wikimedia/parsoid to 0.15.0-a21	mediawiki/vendor	master	+336 -221
Revert part of 55da3090: Setting up template args exposes other bugs	mediawiki/services/parsoid	master	+6 -1

Customize query in gerrit

Related Objects

Mentioned In: T347507: Parsoid doesn't support extensions/modules which crawl the frame stack for arguments
T302007: PHP Notice: Undefined index: mwf247
Mentioned Here: rGPAR55da3090d0e7: Ensure frame args are properly set up in the pipeline

Event Timeline

hashar created this task.Feb 17 2022, 8:43 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 17 2022, 8:43 AM

Minimal reproducer:

$ echo -e '{{fontes|noref=|ref=\n<ref name=Cicero>{{Cicero|deorum|2|7|7|LL|com=mergi}}</ref>}}' | php bin/parse.php --apiURL=https://la.wiktionary.org/w/api.php

Worked on -a19, is broken on -a20.

Issue introduced with commit 55da3090d0e7faa49e40a5bce0f68634fea5fc83

The exact issue was introduced on https://gerrit.wikimedia.org/r/c/mediawiki/services/parsoid/+/759621/9/src/Wt2Html/TT/TemplateHandler.php#1137, where we introduce passing the attributes instead of just a [].
Specifically, it breaks on extensions because of https://gerrit.wikimedia.org/g/mediawiki/services/parsoid/+/e94f523d5db3f90e7d69e7baa9f581895e6d2d9a/src/Wt2Html/Grammar.pegphp#357 - where attribs is a KV and we don't expect it.
However, I don't know how to fix it, because I have no idea how "specific" the issue is:

do we need to handle KVs in the Sanitizer always/in some cases? (The breaking call is here: https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/services/parsoid/+/e94f523d5db3f90e7d69e7baa9f581895e6d2d9a/src/Wt2Html/TT/SanitizerHandler.php#74)
or do we need to drop the "options" attributes in some cases?
or something else?

The first question I would ask here is why are we sanitizing an extension tag token. The sanitizer handler runs after extension rendering, so we must be reinserting tokens into the stream somewhere incorrectly

Krinkle moved this task from Untriaged to Feb 2022 on the Wikimedia-production-error board.Feb 17 2022, 4:43 PM

ssastry mentioned this in T302007: PHP Notice: Undefined index: mwf247.Feb 17 2022, 6:17 PM

Change 763786 had a related patch set uploaded (by Isabelle Hurbain-Palatin; author: Isabelle Hurbain-Palatin):

[mediawiki/services/parsoid@master] Remove shuttling of template argument tokens to the end of the stage

https://gerrit.wikimedia.org/r/763786

gerritbot added a project: Patch-For-Review.Feb 18 2022, 5:10 PM

Change 763796 had a related patch set uploaded (by Subramanya Sastry; author: Subramanya Sastry):

[mediawiki/services/parsoid@master] Revert part of 55da3090: Setting up template args exposes other bugs

https://gerrit.wikimedia.org/r/763796

Change 763796 merged by jenkins-bot:

[mediawiki/services/parsoid@master] Revert part of 55da3090: Setting up template args exposes other bugs

https://gerrit.wikimedia.org/r/763796

Change 763813 had a related patch set uploaded (by Subramanya Sastry; author: Subramanya Sastry):

[mediawiki/vendor@master] Bump wikimedia/parsoid to 0.15.0-a21

https://gerrit.wikimedia.org/r/763813

Change 763813 merged by jenkins-bot:

[mediawiki/vendor@master] Bump wikimedia/parsoid to 0.15.0-a21

https://gerrit.wikimedia.org/r/763813

ssastry closed this task as Resolved.Feb 25 2022, 7:11 PM

Thank you for the fix @ihurbain !

Thank @ssastry - his patch is the correct one :P (We did discuss this thing at length with @Arlolra too :) )

Change 763786 abandoned by Isabelle Hurbain-Palatin:

[mediawiki/services/parsoid@master] Remove shuttling of template argument tokens to the end of the stage

Reason:

Fixed in Ic14a7b77442d8404e5d0c7da6517a3cb81b08359