Page MenuHomePhabricator

Production crashers in WrapSections code because of missing properties ($dsr, $pi, $parts)
Open, HighPublicPRODUCTION ERROR

Description

Error

MediaWiki version: 1.35.0-wmf.10

message
PHP Notice: Undefined property: stdClass::$pi

Impact

Notes

Details

Request ID
XfLGeApAIDYAAB-svV4AAACK
Request URL
https://fr.wikipedia.org/w/rest.php/fr.wikipedia.org/v3/page/pagebundle/Utilisateur%3AWarp3%2Ftest4/165009870
Stack Trace
exception.trace
#0 /srv/deployment/parsoid/deploy-cache/revs/75d72e838eea0e5b7dd04c292539f69d4802e2f8/src/src/Wt2Html/PP/Processors/WrapSections.php(390): MWExceptionHandler::handleError(integer, string, string, integer, array)
#1 /srv/deployment/parsoid/deploy-cache/revs/75d72e838eea0e5b7dd04c292539f69d4802e2f8/src/src/Wt2Html/PP/Processors/WrapSections.php(448): Parsoid\Wt2Html\PP\Processors\WrapSections->resolveTplExtSectionConflicts(array)
#2 /srv/deployment/parsoid/deploy-cache/revs/75d72e838eea0e5b7dd04c292539f69d4802e2f8/src/src/Wt2Html/DOMPostProcessor.php(151): Parsoid\Wt2Html\PP\Processors\WrapSections->run(DOMElement, Parsoid\Config\Env, array, boolean)
#3 /srv/deployment/parsoid/deploy-cache/revs/75d72e838eea0e5b7dd04c292539f69d4802e2f8/src/src/Wt2Html/DOMPostProcessor.php(828): Parsoid\Wt2Html\DOMPostProcessor->Parsoid\Wt2Html\{closure}(DOMElement, Parsoid\Config\Env, array, boolean)
#4 /srv/deployment/parsoid/deploy-cache/revs/75d72e838eea0e5b7dd04c292539f69d4802e2f8/src/src/Wt2Html/DOMPostProcessor.php(881): Parsoid\Wt2Html\DOMPostProcessor->doPostProcess(DOMDocument)
#5 /srv/deployment/parsoid/deploy-cache/revs/75d72e838eea0e5b7dd04c292539f69d4802e2f8/src/src/Wt2Html/DOMPostProcessor.php(898): Parsoid\Wt2Html\DOMPostProcessor->process(DOMDocument)
#6 /srv/deployment/parsoid/deploy-cache/revs/75d72e838eea0e5b7dd04c292539f69d4802e2f8/src/src/Wt2Html/ParserPipeline.php(148): Parsoid\Wt2Html\DOMPostProcessor->processChunkily(string, array)
#7 /srv/deployment/parsoid/deploy-cache/revs/75d72e838eea0e5b7dd04c292539f69d4802e2f8/src/src/Wt2Html/ParserPipeline.php(198): Parsoid\Wt2Html\ParserPipeline->parseChunkily(string, array)
#8 /srv/deployment/parsoid/deploy-cache/revs/75d72e838eea0e5b7dd04c292539f69d4802e2f8/src/src/Wt2Html/ParserPipelineFactory.php(299): Parsoid\Wt2Html\ParserPipeline->parseToplevelDoc(string, array)
#9 /srv/deployment/parsoid/deploy-cache/revs/75d72e838eea0e5b7dd04c292539f69d4802e2f8/src/src/WikitextContentModelHandler.php(78): Parsoid\Wt2Html\ParserPipelineFactory->parse(string)
#10 /srv/deployment/parsoid/deploy-cache/revs/75d72e838eea0e5b7dd04c292539f69d4802e2f8/src/src/Parsoid.php(93): Parsoid\WikitextContentModelHandler->toHTML(Parsoid\Config\Env)
#11 /srv/deployment/parsoid/deploy-cache/revs/75d72e838eea0e5b7dd04c292539f69d4802e2f8/src/src/Parsoid.php(123): Parsoid\Parsoid->parseWikitext(MWParsoid\Config\PageConfig, array)
#12 /srv/deployment/parsoid/deploy-cache/revs/75d72e838eea0e5b7dd04c292539f69d4802e2f8/src/extension/src/Rest/Handler/ParsoidHandler.php(581): Parsoid\Parsoid->wikitext2html(MWParsoid\Config\PageConfig, array, NULL)
#13 /srv/deployment/parsoid/deploy-cache/revs/75d72e838eea0e5b7dd04c292539f69d4802e2f8/src/extension/src/Rest/Handler/PageHandler.php(52): MWParsoid\Rest\Handler\ParsoidHandler->wt2html(Parsoid\Config\Env, array)
#14 /srv/mediawiki/php-1.35.0-wmf.10/includes/Rest/Router.php(314): MWParsoid\Rest\Handler\PageHandler->execute()
#15 /srv/mediawiki/php-1.35.0-wmf.10/includes/Rest/Router.php(285): MediaWiki\Rest\Router->executeHandler(MWParsoid\Rest\Handler\PageHandler)
#16 /srv/mediawiki/php-1.35.0-wmf.10/includes/Rest/EntryPoint.php(111): MediaWiki\Rest\Router->execute(MediaWiki\Rest\RequestFromGlobals)
#17 /srv/mediawiki/php-1.35.0-wmf.10/includes/Rest/EntryPoint.php(78): MediaWiki\Rest\EntryPoint->execute()
#18 /srv/mediawiki/php-1.35.0-wmf.10/rest.php(31): MediaWiki\Rest\EntryPoint::main()
#19 /srv/mediawiki/w/rest.php(3): require(string)
#20 {main}

Event Timeline

Arlolra created this task.Dec 12 2019, 11:39 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptDec 12 2019, 11:39 PM
Arlolra triaged this task as Medium priority.Dec 12 2019, 11:39 PM

Similarly, https://logstash.wikimedia.org/app/kibana#/doc/logstash-*/logstash-2019.12.12/parsoid-php/?id=AW78XHTvh3Uj6x1zaL8G

PHP Notice: Undefined property: stdClass::$parts

#0 /srv/deployment/parsoid/deploy-cache/revs/75d72e838eea0e5b7dd04c292539f69d4802e2f8/src/src/Wt2Html/PP/Processors/WrapSections.php(381): MWExceptionHandler::handleError(integer, string, string, integer, array)
#1 /srv/deployment/parsoid/deploy-cache/revs/75d72e838eea0e5b7dd04c292539f69d4802e2f8/src/src/Wt2Html/PP/Processors/WrapSections.php(448): Parsoid\Wt2Html\PP\Processors\WrapSections->resolveTplExtSectionConflicts(array)
#2 /srv/deployment/parsoid/deploy-cache/revs/75d72e838eea0e5b7dd04c292539f69d4802e2f8/src/src/Wt2Html/DOMPostProcessor.php(151): Parsoid\Wt2Html\PP\Processors\WrapSections->run(DOMElement, Parsoid\Config\Env, array, boolean)
#3 /srv/deployment/parsoid/deploy-cache/revs/75d72e838eea0e5b7dd04c292539f69d4802e2f8/src/src/Wt2Html/DOMPostProcessor.php(828): Parsoid\Wt2Html\DOMPostProcessor->Parsoid\Wt2Html\{closure}(DOMElement, Parsoid\Config\Env, array, boolean)
#4 /srv/deployment/parsoid/deploy-cache/revs/75d72e838eea0e5b7dd04c292539f69d4802e2f8/src/src/Wt2Html/DOMPostProcessor.php(881): Parsoid\Wt2Html\DOMPostProcessor->doPostProcess(DOMDocument)
#5 /srv/deployment/parsoid/deploy-cache/revs/75d72e838eea0e5b7dd04c292539f69d4802e2f8/src/src/Wt2Html/DOMPostProcessor.php(898): Parsoid\Wt2Html\DOMPostProcessor->process(DOMDocument)
#6 /srv/deployment/parsoid/deploy-cache/revs/75d72e838eea0e5b7dd04c292539f69d4802e2f8/src/src/Wt2Html/ParserPipeline.php(148): Parsoid\Wt2Html\DOMPostProcessor->processChunkily(string, array)
#7 /srv/deployment/parsoid/deploy-cache/revs/75d72e838eea0e5b7dd04c292539f69d4802e2f8/src/src/Wt2Html/ParserPipeline.php(198): Parsoid\Wt2Html\ParserPipeline->parseChunkily(string, array)
#8 /srv/deployment/parsoid/deploy-cache/revs/75d72e838eea0e5b7dd04c292539f69d4802e2f8/src/src/Wt2Html/ParserPipelineFactory.php(299): Parsoid\Wt2Html\ParserPipeline->parseToplevelDoc(string, array)
#9 /srv/deployment/parsoid/deploy-cache/revs/75d72e838eea0e5b7dd04c292539f69d4802e2f8/src/src/WikitextContentModelHandler.php(78): Parsoid\Wt2Html\ParserPipelineFactory->parse(string)
#10 /srv/deployment/parsoid/deploy-cache/revs/75d72e838eea0e5b7dd04c292539f69d4802e2f8/src/src/Parsoid.php(93): Parsoid\WikitextContentModelHandler->toHTML(Parsoid\Config\Env)
#11 /srv/deployment/parsoid/deploy-cache/revs/75d72e838eea0e5b7dd04c292539f69d4802e2f8/src/src/Parsoid.php(123): Parsoid\Parsoid->parseWikitext(MWParsoid\Config\PageConfig, array)
#12 /srv/deployment/parsoid/deploy-cache/revs/75d72e838eea0e5b7dd04c292539f69d4802e2f8/src/extension/src/Rest/Handler/ParsoidHandler.php(581): Parsoid\Parsoid->wikitext2html(MWParsoid\Config\PageConfig, array, NULL)
#13 /srv/deployment/parsoid/deploy-cache/revs/75d72e838eea0e5b7dd04c292539f69d4802e2f8/src/extension/src/Rest/Handler/PageHandler.php(52): MWParsoid\Rest\Handler\ParsoidHandler->wt2html(Parsoid\Config\Env, array)
#14 /srv/mediawiki/php-1.35.0-wmf.10/includes/Rest/Router.php(314): MWParsoid\Rest\Handler\PageHandler->execute()
#15 /srv/mediawiki/php-1.35.0-wmf.10/includes/Rest/Router.php(285): MediaWiki\Rest\Router->executeHandler(MWParsoid\Rest\Handler\PageHandler)
#16 /srv/mediawiki/php-1.35.0-wmf.10/includes/Rest/EntryPoint.php(111): MediaWiki\Rest\Router->execute(MediaWiki\Rest\RequestFromGlobals)
#17 /srv/mediawiki/php-1.35.0-wmf.10/includes/Rest/EntryPoint.php(78): MediaWiki\Rest\EntryPoint->execute()
#18 /srv/mediawiki/php-1.35.0-wmf.10/rest.php(31): MediaWiki\Rest\EntryPoint::main()
#19 /srv/mediawiki/w/rest.php(3): require(string)
#20 {main}
Aklapper edited projects, added Parsoid; removed Parsoid-PHP.Apr 10 2020, 4:27 PM
ssastry moved this task from Needs Triage to Bugs & Crashers on the Parsoid board.Apr 10 2020, 4:48 PM
brennen moved this task from Backlog to Logs/Train on the User-brennen board.
mmodell added a subscriber: mmodell.

Seen again:

message
PHP Notice: Undefined property: stdClass::$parts
exception.trace
#0 /srv/mediawiki/php-1.35.0-wmf.39/vendor/wikimedia/parsoid/src/Wt2Html/PP/Processors/WrapSections.php(380): MWExceptionHandler::handleError(integer, string, string, integer, array)
#1 /srv/mediawiki/php-1.35.0-wmf.39/vendor/wikimedia/parsoid/src/Wt2Html/PP/Processors/WrapSections.php(447): Wikimedia\Parsoid\Wt2Html\PP\Processors\WrapSections->resolveTplExtSectionConflicts(array)
#2 /srv/mediawiki/php-1.35.0-wmf.39/vendor/wikimedia/parsoid/src/Wt2Html/DOMPostProcessor.php(156): Wikimedia\Parsoid\Wt2Html\PP\Processors\WrapSections->run(Wikimedia\Parsoid\Config\Env, DOMElement, array, boolean)
#3 /srv/mediawiki/php-1.35.0-wmf.39/vendor/wikimedia/parsoid/src/Wt2Html/DOMPostProcessor.php(856): Wikimedia\Parsoid\Wt2Html\DOMPostProcessor->Wikimedia\Parsoid\Wt2Html\{closure}(DOMElement, array, boolean)
#4 /srv/mediawiki/php-1.35.0-wmf.39/vendor/wikimedia/parsoid/src/Wt2Html/DOMPostProcessor.php(905): Wikimedia\Parsoid\Wt2Html\DOMPostProcessor->doPostProcess(DOMDocument)
#5 /srv/mediawiki/php-1.35.0-wmf.39/vendor/wikimedia/parsoid/src/Wt2Html/DOMPostProcessor.php(922): Wikimedia\Parsoid\Wt2Html\DOMPostProcessor->process(DOMDocument)
#6 /srv/mediawiki/php-1.35.0-wmf.39/vendor/wikimedia/parsoid/src/Wt2Html/ParserPipeline.php(152): Wikimedia\Parsoid\Wt2Html\DOMPostProcessor->processChunkily(string, array)
#7 /srv/mediawiki/php-1.35.0-wmf.39/vendor/wikimedia/parsoid/src/Wt2Html/ParserPipeline.php(202): Wikimedia\Parsoid\Wt2Html\ParserPipeline->parseChunkily(string, array)
#8 /srv/mediawiki/php-1.35.0-wmf.39/vendor/wikimedia/parsoid/src/Wt2Html/ParserPipelineFactory.php(299): Wikimedia\Parsoid\Wt2Html\ParserPipeline->parseToplevelDoc(string, array)
#9 /srv/mediawiki/php-1.35.0-wmf.39/vendor/wikimedia/parsoid/src/Core/WikitextContentModelHandler.php(78): Wikimedia\Parsoid\Wt2Html\ParserPipelineFactory->parse(string)
#10 /srv/mediawiki/php-1.35.0-wmf.39/vendor/wikimedia/parsoid/src/Parsoid.php(152): Wikimedia\Parsoid\Core\WikitextContentModelHandler->toDOM(Wikimedia\Parsoid\Config\Env)
#11 /srv/mediawiki/php-1.35.0-wmf.39/vendor/wikimedia/parsoid/src/Parsoid.php(184): Wikimedia\Parsoid\Parsoid->parseWikitext(MWParsoid\Config\PageConfig, array)
#12 /srv/mediawiki/php-1.35.0-wmf.39/vendor/wikimedia/parsoid/extension/src/Rest/Handler/ParsoidHandler.php(533): Wikimedia\Parsoid\Parsoid->wikitext2html(MWParsoid\Config\PageConfig, array, NULL)
#13 /srv/mediawiki/php-1.35.0-wmf.39/vendor/wikimedia/parsoid/extension/src/Rest/Handler/PageHandler.php(66): MWParsoid\Rest\Handler\ParsoidHandler->wt2html(MWParsoid\Config\PageConfig, array)
#14 /srv/mediawiki/php-1.35.0-wmf.39/includes/Rest/Router.php(362): MWParsoid\Rest\Handler\PageHandler->execute()
#15 /srv/mediawiki/php-1.35.0-wmf.39/includes/Rest/Router.php(317): MediaWiki\Rest\Router->executeHandler(MWParsoid\Rest\Handler\PageHandler)
#16 /srv/mediawiki/php-1.35.0-wmf.39/includes/Rest/EntryPoint.php(139): MediaWiki\Rest\Router->execute(MediaWiki\Rest\RequestFromGlobals)
#17 /srv/mediawiki/php-1.35.0-wmf.39/includes/Rest/EntryPoint.php(106): MediaWiki\Rest\EntryPoint->execute()
#18 /srv/mediawiki/php-1.35.0-wmf.39/rest.php(31): MediaWiki\Rest\EntryPoint::main()
#19 /srv/mediawiki/w/rest.php(3): require(string)
#20 {main}
ssastry renamed this task from PHP Notice: Undefined property: stdClass::$pi to Production crashers in WrapSections code because of missing properties ($dsr, $pi, $parts).Jul 22 2020, 7:00 PM
ssastry added a subscriber: jeena.
ssastry raised the priority of this task from Medium to High.Aug 12 2020, 9:59 PM
ssastry added a subscriber: hashar.

So, this below is strange. The curl request works on scandium, but not on production eqiad (wtp1025) or codfw (wtp1040) servers. As of now (when I tested this), scandium and production have the exact same code.

ssastry@scandium:~$ curl -v -x scandium.eqiad.wmnet:80 http://en.wikipedia.org/w/rest.php/en.wikipedia.org/v3/page/pagebundle/User_talk%3AEzekiel_7%3A19/966442756 > /dev/null
...
< HTTP/1.1 200 OK
....
ssastry@scandium:~$ curl -v -x wtp1025.eqiad.wmnet:80 http://en.wikipedia.org/w/rest.php/en.wikipedia.org/v3/page/pagebundle/User_talk%3AEzekiel_7%3A19/966442756 > /dev/null
...
< HTTP/1.1 500 Internal Server Error
...
ssastry@scandium:~$ curl -v -x wtp1040.eqiad.wmnet:80 http://en.wikipedia.org/w/rest.php/en.wikipedia.org/v3/page/pagebundle/User_talk%3AEzekiel_7%3A19/966442756 > /dev/null
...
< HTTP/1.1 500 Internal Server Error
...

I cannot reproduce it on the commandline locally on my laptop or on scandium. That makes it tricky. So, I'll try to (a) either look at other reported crashing pages (b) try to progressively trim this enwiki talk page and see if I can find a reduced test case.

I managed to reduce the page to: https://en.wikipedia.org/w/index.php?title=User:SSastry_(WMF)/sandbox/T240642&oldid=972593962 which still crashes on wtp1025 ... to be continued.

So, it is now reduced to this page content:

{{User_talk:Ezekiel_7:19/Header}}
== A ==

curl -v -x wtp1040.eqiad.wmnet:80 'http://en.wikipedia.org/w/rest.php/en.wikipedia.org/v3/page/html/User:SSastry_(WMF)%2Fsandbox%2FT240642/972596509' > /dev/null now fails.

And, based on examining the output on revid 972596588 which doesn't crash, I determined that scandium has $wgParsoidSettings['rtTestMode'] = true; enabled which changes output ever so slightly by leaving behind additional information about deleted tags which probably lets WrapSections pass gather more accurate DSR information. So, that is already a good discovery and that means we should look at when these tags are stripped relative to the WrapSections pass.

Note this task was filed with http://fr.wikipedia.org/w/rest.php/fr.wikipedia.org/v3/page/pagebundle/Utilisateur%3AWarp3%2Ftest4/165009870 which no more causes a 500 on wtp1040.eqiad.wmnet.

When I looked at the log after 1.36.0-wmf.4 deployment, the issue only appeared on the Ezekiel_7 page. So it is most probably be a corner case that triggers the issue.

Another related error:

[{exception_id}] {exception_url} ErrorException from line 255 of /srv/mediawiki/php-1.36.0-wmf.5/vendor/wikimedia/parsoid/src/Wt2Html/PP/Processors/WrapSections.php: PHP Notice: Undefined property: stdClass::$dsr

(I apologize if that's unhelpful, just trying to be thorough)

dancy added a subscriber: dancy.Sep 23 2020, 7:48 PM

360 of these (PHP Notice: Undefined property: stdClass::$dsr) in the last 4 hours.

The error is reproducible with this snippet from frwiki. Everything is required including the extra newlines.

<noinclude>
{{Documentation}}
<noinclude>


</noinclude>
</noinclude>

See CLI session below.

[subbu@earth:~/work/wmf/parsoid] php bin/parse.php --domain fr.wikipedia.org --wrapSections < /tmp/frwt > /dev/null
[error] Do not have necessary info. to encapsulate Tpl: 0
Start Elt : <meta typeof="mw:Transclusion" about="#mwt1" data-object-id="31"/>
End Elt   : <meta typeof="mw:Transclusion/End" about="#mwt1" data-object-id="250"/>
Start DSR : [42,29,0,0]
End   DSR : [null,29,null,null]
hashar removed a subscriber: hashar.Oct 14 2020, 7:20 PM

Change 634118 had a related patch set uploaded (by Subramanya Sastry; owner: Subramanya Sastry):
[mediawiki/services/parsoid@master] Fix broken nested <noinclude> handling

https://gerrit.wikimedia.org/r/634118

Change 634118 merged by jenkins-bot:
[mediawiki/services/parsoid@master] Fix broken nested <noinclude> handling

https://gerrit.wikimedia.org/r/634118

Change 635100 had a related patch set uploaded (by C. Scott Ananian; owner: C. Scott Ananian):
[mediawiki/vendor@master] Bump wikimedia/parsoid to 0.13.0-a12

https://gerrit.wikimedia.org/r/635100

Change 635100 merged by jenkins-bot:
[mediawiki/vendor@master] Bump wikimedia/parsoid to 0.13.0-a12

https://gerrit.wikimedia.org/r/635100

dancy removed a subscriber: dancy.Oct 20 2020, 3:07 PM

Additional test titles (seen in wmf14):

  • /w/rest.php/en.wikipedia.org/v3/page/pagebundle/Utah_Warriors/979957184

FYI, seeing more of these following 1.36.0-wmf.22 all wikis deployment (T267415).

Noticing what feels like an uptick in these in 1.36.0-wmf.28.

Change 662804 had a related patch set uploaded (by Subramanya Sastry; owner: Subramanya Sastry):
[mediawiki/services/parsoid@master] WrapSections: Don't crash if we have incomplete DSR information

https://gerrit.wikimedia.org/r/662804

Change 662804 merged by jenkins-bot:
[mediawiki/services/parsoid@master] WrapSections: Don't crash if we have incomplete DSR information

https://gerrit.wikimedia.org/r/662804