Page MenuHomePhabricator

Production crashers in WrapSections code because of missing properties ($dsr, $pi, $parts)
Closed, ResolvedPublicPRODUCTION ERROR

Description

Error

MediaWiki version: 1.35.0-wmf.10

message
PHP Notice: Undefined property: stdClass::$pi

Impact

Notes

Details

Request ID
XfLGeApAIDYAAB-svV4AAACK
Request URL
https://fr.wikipedia.org/w/rest.php/fr.wikipedia.org/v3/page/pagebundle/Utilisateur%3AWarp3%2Ftest4/165009870
Stack Trace
exception.trace
#0 /srv/deployment/parsoid/deploy-cache/revs/75d72e838eea0e5b7dd04c292539f69d4802e2f8/src/src/Wt2Html/PP/Processors/WrapSections.php(390): MWExceptionHandler::handleError(integer, string, string, integer, array)
#1 /srv/deployment/parsoid/deploy-cache/revs/75d72e838eea0e5b7dd04c292539f69d4802e2f8/src/src/Wt2Html/PP/Processors/WrapSections.php(448): Parsoid\Wt2Html\PP\Processors\WrapSections->resolveTplExtSectionConflicts(array)
#2 /srv/deployment/parsoid/deploy-cache/revs/75d72e838eea0e5b7dd04c292539f69d4802e2f8/src/src/Wt2Html/DOMPostProcessor.php(151): Parsoid\Wt2Html\PP\Processors\WrapSections->run(DOMElement, Parsoid\Config\Env, array, boolean)
#3 /srv/deployment/parsoid/deploy-cache/revs/75d72e838eea0e5b7dd04c292539f69d4802e2f8/src/src/Wt2Html/DOMPostProcessor.php(828): Parsoid\Wt2Html\DOMPostProcessor->Parsoid\Wt2Html\{closure}(DOMElement, Parsoid\Config\Env, array, boolean)
#4 /srv/deployment/parsoid/deploy-cache/revs/75d72e838eea0e5b7dd04c292539f69d4802e2f8/src/src/Wt2Html/DOMPostProcessor.php(881): Parsoid\Wt2Html\DOMPostProcessor->doPostProcess(DOMDocument)
#5 /srv/deployment/parsoid/deploy-cache/revs/75d72e838eea0e5b7dd04c292539f69d4802e2f8/src/src/Wt2Html/DOMPostProcessor.php(898): Parsoid\Wt2Html\DOMPostProcessor->process(DOMDocument)
#6 /srv/deployment/parsoid/deploy-cache/revs/75d72e838eea0e5b7dd04c292539f69d4802e2f8/src/src/Wt2Html/ParserPipeline.php(148): Parsoid\Wt2Html\DOMPostProcessor->processChunkily(string, array)
#7 /srv/deployment/parsoid/deploy-cache/revs/75d72e838eea0e5b7dd04c292539f69d4802e2f8/src/src/Wt2Html/ParserPipeline.php(198): Parsoid\Wt2Html\ParserPipeline->parseChunkily(string, array)
#8 /srv/deployment/parsoid/deploy-cache/revs/75d72e838eea0e5b7dd04c292539f69d4802e2f8/src/src/Wt2Html/ParserPipelineFactory.php(299): Parsoid\Wt2Html\ParserPipeline->parseToplevelDoc(string, array)
#9 /srv/deployment/parsoid/deploy-cache/revs/75d72e838eea0e5b7dd04c292539f69d4802e2f8/src/src/WikitextContentModelHandler.php(78): Parsoid\Wt2Html\ParserPipelineFactory->parse(string)
#10 /srv/deployment/parsoid/deploy-cache/revs/75d72e838eea0e5b7dd04c292539f69d4802e2f8/src/src/Parsoid.php(93): Parsoid\WikitextContentModelHandler->toHTML(Parsoid\Config\Env)
#11 /srv/deployment/parsoid/deploy-cache/revs/75d72e838eea0e5b7dd04c292539f69d4802e2f8/src/src/Parsoid.php(123): Parsoid\Parsoid->parseWikitext(MWParsoid\Config\PageConfig, array)
#12 /srv/deployment/parsoid/deploy-cache/revs/75d72e838eea0e5b7dd04c292539f69d4802e2f8/src/extension/src/Rest/Handler/ParsoidHandler.php(581): Parsoid\Parsoid->wikitext2html(MWParsoid\Config\PageConfig, array, NULL)
#13 /srv/deployment/parsoid/deploy-cache/revs/75d72e838eea0e5b7dd04c292539f69d4802e2f8/src/extension/src/Rest/Handler/PageHandler.php(52): MWParsoid\Rest\Handler\ParsoidHandler->wt2html(Parsoid\Config\Env, array)
#14 /srv/mediawiki/php-1.35.0-wmf.10/includes/Rest/Router.php(314): MWParsoid\Rest\Handler\PageHandler->execute()
#15 /srv/mediawiki/php-1.35.0-wmf.10/includes/Rest/Router.php(285): MediaWiki\Rest\Router->executeHandler(MWParsoid\Rest\Handler\PageHandler)
#16 /srv/mediawiki/php-1.35.0-wmf.10/includes/Rest/EntryPoint.php(111): MediaWiki\Rest\Router->execute(MediaWiki\Rest\RequestFromGlobals)
#17 /srv/mediawiki/php-1.35.0-wmf.10/includes/Rest/EntryPoint.php(78): MediaWiki\Rest\EntryPoint->execute()
#18 /srv/mediawiki/php-1.35.0-wmf.10/rest.php(31): MediaWiki\Rest\EntryPoint::main()
#19 /srv/mediawiki/w/rest.php(3): require(string)
#20 {main}

Event Timeline

Arlolra triaged this task as Medium priority.Dec 12 2019, 11:39 PM

Similarly, https://logstash.wikimedia.org/app/kibana#/doc/logstash-*/logstash-2019.12.12/parsoid-php/?id=AW78XHTvh3Uj6x1zaL8G

PHP Notice: Undefined property: stdClass::$parts

#0 /srv/deployment/parsoid/deploy-cache/revs/75d72e838eea0e5b7dd04c292539f69d4802e2f8/src/src/Wt2Html/PP/Processors/WrapSections.php(381): MWExceptionHandler::handleError(integer, string, string, integer, array)
#1 /srv/deployment/parsoid/deploy-cache/revs/75d72e838eea0e5b7dd04c292539f69d4802e2f8/src/src/Wt2Html/PP/Processors/WrapSections.php(448): Parsoid\Wt2Html\PP\Processors\WrapSections->resolveTplExtSectionConflicts(array)
#2 /srv/deployment/parsoid/deploy-cache/revs/75d72e838eea0e5b7dd04c292539f69d4802e2f8/src/src/Wt2Html/DOMPostProcessor.php(151): Parsoid\Wt2Html\PP\Processors\WrapSections->run(DOMElement, Parsoid\Config\Env, array, boolean)
#3 /srv/deployment/parsoid/deploy-cache/revs/75d72e838eea0e5b7dd04c292539f69d4802e2f8/src/src/Wt2Html/DOMPostProcessor.php(828): Parsoid\Wt2Html\DOMPostProcessor->Parsoid\Wt2Html\{closure}(DOMElement, Parsoid\Config\Env, array, boolean)
#4 /srv/deployment/parsoid/deploy-cache/revs/75d72e838eea0e5b7dd04c292539f69d4802e2f8/src/src/Wt2Html/DOMPostProcessor.php(881): Parsoid\Wt2Html\DOMPostProcessor->doPostProcess(DOMDocument)
#5 /srv/deployment/parsoid/deploy-cache/revs/75d72e838eea0e5b7dd04c292539f69d4802e2f8/src/src/Wt2Html/DOMPostProcessor.php(898): Parsoid\Wt2Html\DOMPostProcessor->process(DOMDocument)
#6 /srv/deployment/parsoid/deploy-cache/revs/75d72e838eea0e5b7dd04c292539f69d4802e2f8/src/src/Wt2Html/ParserPipeline.php(148): Parsoid\Wt2Html\DOMPostProcessor->processChunkily(string, array)
#7 /srv/deployment/parsoid/deploy-cache/revs/75d72e838eea0e5b7dd04c292539f69d4802e2f8/src/src/Wt2Html/ParserPipeline.php(198): Parsoid\Wt2Html\ParserPipeline->parseChunkily(string, array)
#8 /srv/deployment/parsoid/deploy-cache/revs/75d72e838eea0e5b7dd04c292539f69d4802e2f8/src/src/Wt2Html/ParserPipelineFactory.php(299): Parsoid\Wt2Html\ParserPipeline->parseToplevelDoc(string, array)
#9 /srv/deployment/parsoid/deploy-cache/revs/75d72e838eea0e5b7dd04c292539f69d4802e2f8/src/src/WikitextContentModelHandler.php(78): Parsoid\Wt2Html\ParserPipelineFactory->parse(string)
#10 /srv/deployment/parsoid/deploy-cache/revs/75d72e838eea0e5b7dd04c292539f69d4802e2f8/src/src/Parsoid.php(93): Parsoid\WikitextContentModelHandler->toHTML(Parsoid\Config\Env)
#11 /srv/deployment/parsoid/deploy-cache/revs/75d72e838eea0e5b7dd04c292539f69d4802e2f8/src/src/Parsoid.php(123): Parsoid\Parsoid->parseWikitext(MWParsoid\Config\PageConfig, array)
#12 /srv/deployment/parsoid/deploy-cache/revs/75d72e838eea0e5b7dd04c292539f69d4802e2f8/src/extension/src/Rest/Handler/ParsoidHandler.php(581): Parsoid\Parsoid->wikitext2html(MWParsoid\Config\PageConfig, array, NULL)
#13 /srv/deployment/parsoid/deploy-cache/revs/75d72e838eea0e5b7dd04c292539f69d4802e2f8/src/extension/src/Rest/Handler/PageHandler.php(52): MWParsoid\Rest\Handler\ParsoidHandler->wt2html(Parsoid\Config\Env, array)
#14 /srv/mediawiki/php-1.35.0-wmf.10/includes/Rest/Router.php(314): MWParsoid\Rest\Handler\PageHandler->execute()
#15 /srv/mediawiki/php-1.35.0-wmf.10/includes/Rest/Router.php(285): MediaWiki\Rest\Router->executeHandler(MWParsoid\Rest\Handler\PageHandler)
#16 /srv/mediawiki/php-1.35.0-wmf.10/includes/Rest/EntryPoint.php(111): MediaWiki\Rest\Router->execute(MediaWiki\Rest\RequestFromGlobals)
#17 /srv/mediawiki/php-1.35.0-wmf.10/includes/Rest/EntryPoint.php(78): MediaWiki\Rest\EntryPoint->execute()
#18 /srv/mediawiki/php-1.35.0-wmf.10/rest.php(31): MediaWiki\Rest\EntryPoint::main()
#19 /srv/mediawiki/w/rest.php(3): require(string)
#20 {main}
mmodell added a subscriber: mmodell.

Seen again:

message
PHP Notice: Undefined property: stdClass::$parts
exception.trace
#0 /srv/mediawiki/php-1.35.0-wmf.39/vendor/wikimedia/parsoid/src/Wt2Html/PP/Processors/WrapSections.php(380): MWExceptionHandler::handleError(integer, string, string, integer, array)
#1 /srv/mediawiki/php-1.35.0-wmf.39/vendor/wikimedia/parsoid/src/Wt2Html/PP/Processors/WrapSections.php(447): Wikimedia\Parsoid\Wt2Html\PP\Processors\WrapSections->resolveTplExtSectionConflicts(array)
#2 /srv/mediawiki/php-1.35.0-wmf.39/vendor/wikimedia/parsoid/src/Wt2Html/DOMPostProcessor.php(156): Wikimedia\Parsoid\Wt2Html\PP\Processors\WrapSections->run(Wikimedia\Parsoid\Config\Env, DOMElement, array, boolean)
#3 /srv/mediawiki/php-1.35.0-wmf.39/vendor/wikimedia/parsoid/src/Wt2Html/DOMPostProcessor.php(856): Wikimedia\Parsoid\Wt2Html\DOMPostProcessor->Wikimedia\Parsoid\Wt2Html\{closure}(DOMElement, array, boolean)
#4 /srv/mediawiki/php-1.35.0-wmf.39/vendor/wikimedia/parsoid/src/Wt2Html/DOMPostProcessor.php(905): Wikimedia\Parsoid\Wt2Html\DOMPostProcessor->doPostProcess(DOMDocument)
#5 /srv/mediawiki/php-1.35.0-wmf.39/vendor/wikimedia/parsoid/src/Wt2Html/DOMPostProcessor.php(922): Wikimedia\Parsoid\Wt2Html\DOMPostProcessor->process(DOMDocument)
#6 /srv/mediawiki/php-1.35.0-wmf.39/vendor/wikimedia/parsoid/src/Wt2Html/ParserPipeline.php(152): Wikimedia\Parsoid\Wt2Html\DOMPostProcessor->processChunkily(string, array)
#7 /srv/mediawiki/php-1.35.0-wmf.39/vendor/wikimedia/parsoid/src/Wt2Html/ParserPipeline.php(202): Wikimedia\Parsoid\Wt2Html\ParserPipeline->parseChunkily(string, array)
#8 /srv/mediawiki/php-1.35.0-wmf.39/vendor/wikimedia/parsoid/src/Wt2Html/ParserPipelineFactory.php(299): Wikimedia\Parsoid\Wt2Html\ParserPipeline->parseToplevelDoc(string, array)
#9 /srv/mediawiki/php-1.35.0-wmf.39/vendor/wikimedia/parsoid/src/Core/WikitextContentModelHandler.php(78): Wikimedia\Parsoid\Wt2Html\ParserPipelineFactory->parse(string)
#10 /srv/mediawiki/php-1.35.0-wmf.39/vendor/wikimedia/parsoid/src/Parsoid.php(152): Wikimedia\Parsoid\Core\WikitextContentModelHandler->toDOM(Wikimedia\Parsoid\Config\Env)
#11 /srv/mediawiki/php-1.35.0-wmf.39/vendor/wikimedia/parsoid/src/Parsoid.php(184): Wikimedia\Parsoid\Parsoid->parseWikitext(MWParsoid\Config\PageConfig, array)
#12 /srv/mediawiki/php-1.35.0-wmf.39/vendor/wikimedia/parsoid/extension/src/Rest/Handler/ParsoidHandler.php(533): Wikimedia\Parsoid\Parsoid->wikitext2html(MWParsoid\Config\PageConfig, array, NULL)
#13 /srv/mediawiki/php-1.35.0-wmf.39/vendor/wikimedia/parsoid/extension/src/Rest/Handler/PageHandler.php(66): MWParsoid\Rest\Handler\ParsoidHandler->wt2html(MWParsoid\Config\PageConfig, array)
#14 /srv/mediawiki/php-1.35.0-wmf.39/includes/Rest/Router.php(362): MWParsoid\Rest\Handler\PageHandler->execute()
#15 /srv/mediawiki/php-1.35.0-wmf.39/includes/Rest/Router.php(317): MediaWiki\Rest\Router->executeHandler(MWParsoid\Rest\Handler\PageHandler)
#16 /srv/mediawiki/php-1.35.0-wmf.39/includes/Rest/EntryPoint.php(139): MediaWiki\Rest\Router->execute(MediaWiki\Rest\RequestFromGlobals)
#17 /srv/mediawiki/php-1.35.0-wmf.39/includes/Rest/EntryPoint.php(106): MediaWiki\Rest\EntryPoint->execute()
#18 /srv/mediawiki/php-1.35.0-wmf.39/rest.php(31): MediaWiki\Rest\EntryPoint::main()
#19 /srv/mediawiki/w/rest.php(3): require(string)
#20 {main}
ssastry renamed this task from PHP Notice: Undefined property: stdClass::$pi to Production crashers in WrapSections code because of missing properties ($dsr, $pi, $parts).Jul 22 2020, 7:00 PM
ssastry added a subscriber: jeena.
ssastry raised the priority of this task from Medium to High.Aug 12 2020, 9:59 PM
ssastry added a subscriber: hashar.

So, this below is strange. The curl request works on scandium, but not on production eqiad (wtp1025) or codfw (wtp1040) servers. As of now (when I tested this), scandium and production have the exact same code.

ssastry@scandium:~$ curl -v -x scandium.eqiad.wmnet:80 http://en.wikipedia.org/w/rest.php/en.wikipedia.org/v3/page/pagebundle/User_talk%3AEzekiel_7%3A19/966442756 > /dev/null
...
< HTTP/1.1 200 OK
....
ssastry@scandium:~$ curl -v -x wtp1025.eqiad.wmnet:80 http://en.wikipedia.org/w/rest.php/en.wikipedia.org/v3/page/pagebundle/User_talk%3AEzekiel_7%3A19/966442756 > /dev/null
...
< HTTP/1.1 500 Internal Server Error
...
ssastry@scandium:~$ curl -v -x wtp1040.eqiad.wmnet:80 http://en.wikipedia.org/w/rest.php/en.wikipedia.org/v3/page/pagebundle/User_talk%3AEzekiel_7%3A19/966442756 > /dev/null
...
< HTTP/1.1 500 Internal Server Error
...

I cannot reproduce it on the commandline locally on my laptop or on scandium. That makes it tricky. So, I'll try to (a) either look at other reported crashing pages (b) try to progressively trim this enwiki talk page and see if I can find a reduced test case.

So, it is now reduced to this page content:

{{User_talk:Ezekiel_7:19/Header}}
== A ==

curl -v -x wtp1040.eqiad.wmnet:80 'http://en.wikipedia.org/w/rest.php/en.wikipedia.org/v3/page/html/User:SSastry_(WMF)%2Fsandbox%2FT240642/972596509' > /dev/null now fails.

And, based on examining the output on revid 972596588 which doesn't crash, I determined that scandium has $wgParsoidSettings['rtTestMode'] = true; enabled which changes output ever so slightly by leaving behind additional information about deleted tags which probably lets WrapSections pass gather more accurate DSR information. So, that is already a good discovery and that means we should look at when these tags are stripped relative to the WrapSections pass.

Note this task was filed with http://fr.wikipedia.org/w/rest.php/fr.wikipedia.org/v3/page/pagebundle/Utilisateur%3AWarp3%2Ftest4/165009870 which no more causes a 500 on wtp1040.eqiad.wmnet.

When I looked at the log after 1.36.0-wmf.4 deployment, the issue only appeared on the Ezekiel_7 page. So it is most probably be a corner case that triggers the issue.

Another related error:

[{exception_id}] {exception_url} ErrorException from line 255 of /srv/mediawiki/php-1.36.0-wmf.5/vendor/wikimedia/parsoid/src/Wt2Html/PP/Processors/WrapSections.php: PHP Notice: Undefined property: stdClass::$dsr

(I apologize if that's unhelpful, just trying to be thorough)

360 of these (PHP Notice: Undefined property: stdClass::$dsr) in the last 4 hours.

The error is reproducible with this snippet from frwiki. Everything is required including the extra newlines.

<noinclude>
{{Documentation}}
<noinclude>


</noinclude>
</noinclude>

See CLI session below.

[subbu@earth:~/work/wmf/parsoid] php bin/parse.php --domain fr.wikipedia.org --wrapSections < /tmp/frwt > /dev/null
[error] Do not have necessary info. to encapsulate Tpl: 0
Start Elt : <meta typeof="mw:Transclusion" about="#mwt1" data-object-id="31"/>
End Elt   : <meta typeof="mw:Transclusion/End" about="#mwt1" data-object-id="250"/>
Start DSR : [42,29,0,0]
End   DSR : [null,29,null,null]

Change 634118 had a related patch set uploaded (by Subramanya Sastry; owner: Subramanya Sastry):
[mediawiki/services/parsoid@master] Fix broken nested <noinclude> handling

https://gerrit.wikimedia.org/r/634118

Change 634118 merged by jenkins-bot:
[mediawiki/services/parsoid@master] Fix broken nested <noinclude> handling

https://gerrit.wikimedia.org/r/634118

Change 635100 had a related patch set uploaded (by C. Scott Ananian; owner: C. Scott Ananian):
[mediawiki/vendor@master] Bump wikimedia/parsoid to 0.13.0-a12

https://gerrit.wikimedia.org/r/635100

Change 635100 merged by jenkins-bot:
[mediawiki/vendor@master] Bump wikimedia/parsoid to 0.13.0-a12

https://gerrit.wikimedia.org/r/635100

Additional test titles (seen in wmf14):

  • /w/rest.php/en.wikipedia.org/v3/page/pagebundle/Utah_Warriors/979957184

FYI, seeing more of these following 1.36.0-wmf.22 all wikis deployment (T267415).

Noticing what feels like an uptick in these in 1.36.0-wmf.28.

Change 662804 had a related patch set uploaded (by Subramanya Sastry; owner: Subramanya Sastry):
[mediawiki/services/parsoid@master] WrapSections: Don't crash if we have incomplete DSR information

https://gerrit.wikimedia.org/r/662804

Change 662804 merged by jenkins-bot:
[mediawiki/services/parsoid@master] WrapSections: Don't crash if we have incomplete DSR information

https://gerrit.wikimedia.org/r/662804

Change 667711 had a related patch set uploaded (by Subramanya Sastry; owner: Subramanya Sastry):
[mediawiki/vendor@master] Bump wikimedia/parsoid to 0.13.0-a26

https://gerrit.wikimedia.org/r/667711

Change 667711 merged by jenkins-bot:
[mediawiki/vendor@master] Bump wikimedia/parsoid to 0.13.0-a26

https://gerrit.wikimedia.org/r/667711

I am going to optimistically mark this resolved. Let us reopen if we see newer instances of any of these errors.