Page MenuHomePhabricator

Class 'LathMathML' not found
Closed, DuplicatePublicPRODUCTION ERROR

Description

Error

MediaWiki version: 1.36.0-wmf.11

message
Class 'LathMathML' not found

Impact

train blocker, large volume of errors in production

Notes

Details

Request ID
920b444c-ae68-4573-a4cf-61cd5572479c
Request URL
https://commons.wikimedia.org/w/api.php
Stack Trace
exception.trace
#0 /srv/mediawiki/php-1.36.0-wmf.11/includes/HookContainer/HookContainer.php(333): MathHooks::onParserAfterTidy(Parser, string)
#1 /srv/mediawiki/php-1.36.0-wmf.11/includes/HookContainer/HookContainer.php(140): MediaWiki\HookContainer\HookContainer->callLegacyHook(string, array, array, array)
#2 /srv/mediawiki/php-1.36.0-wmf.11/includes/HookContainer/HookRunner.php(2879): MediaWiki\HookContainer\HookContainer->run(string, array)
#3 /srv/mediawiki/php-1.36.0-wmf.11/includes/parser/Parser.php(1680): MediaWiki\HookContainer\HookRunner->onParserAfterTidy(Parser, string)
#4 /srv/mediawiki/php-1.36.0-wmf.11/includes/parser/Parser.php(649): Parser->internalParseHalfParsed(string, boolean, boolean)
#5 /srv/mediawiki/php-1.36.0-wmf.11/includes/content/WikitextContent.php(374): Parser->parse(string, Title, ParserOptions, boolean, boolean, integer)
#6 /srv/mediawiki/php-1.36.0-wmf.11/includes/content/AbstractContent.php(590): WikitextContent->fillParserOutput(Title, integer, ParserOptions, boolean, ParserOutput)
#7 /srv/mediawiki/php-1.36.0-wmf.11/includes/Revision/RenderedRevision.php(266): AbstractContent->getParserOutput(Title, integer, ParserOptions, boolean)
#8 /srv/mediawiki/php-1.36.0-wmf.11/includes/Revision/RenderedRevision.php(235): MediaWiki\Revision\RenderedRevision->getSlotParserOutputUncached(WikitextContent, boolean)
#9 /srv/mediawiki/php-1.36.0-wmf.11/includes/Revision/RevisionRenderer.php(230): MediaWiki\Revision\RenderedRevision->getSlotParserOutput(string, array)
#10 /srv/mediawiki/php-1.36.0-wmf.11/includes/Revision/RevisionRenderer.php(152): MediaWiki\Revision\RevisionRenderer->combineSlotOutput(MediaWiki\Revision\RenderedRevision, array)
#11 [internal function]: MediaWiki\Revision\RevisionRenderer->MediaWiki\Revision\{closure}(MediaWiki\Revision\RenderedRevision, array)
#12 /srv/mediawiki/php-1.36.0-wmf.11/includes/Revision/RenderedRevision.php(197): call_user_func(Closure, MediaWiki\Revision\RenderedRevision, array)
#13 /srv/mediawiki/php-1.36.0-wmf.11/includes/poolcounter/PoolWorkArticleView.php(216): MediaWiki\Revision\RenderedRevision->getRevisionParserOutput()
#14 /srv/mediawiki/php-1.36.0-wmf.11/includes/poolcounter/PoolCounterWork.php(162): PoolWorkArticleView->doWork()
#15 /srv/mediawiki/php-1.36.0-wmf.11/includes/page/WikiPage.php(1221): PoolCounterWork->execute()
#16 /srv/mediawiki/php-1.36.0-wmf.11/includes/api/ApiParse.php(83): WikiPage->getParserOutput(ParserOptions, integer, boolean)
#17 /srv/mediawiki/php-1.36.0-wmf.11/includes/poolcounter/PoolCounterWorkViaCallback.php(74): ApiParse->{closure}()
#18 /srv/mediawiki/php-1.36.0-wmf.11/includes/poolcounter/PoolCounterWork.php(162): PoolCounterWorkViaCallback->doWork()
#19 /srv/mediawiki/php-1.36.0-wmf.11/includes/api/ApiParse.php(90): PoolCounterWork->execute()
#20 /srv/mediawiki/php-1.36.0-wmf.11/includes/api/ApiParse.php(658): ApiParse->getPageParserOutput(WikiFilePage, NULL, ParserOptions, boolean)
#21 /srv/mediawiki/php-1.36.0-wmf.11/includes/api/ApiParse.php(222): ApiParse->getParsedContent(WikiFilePage, ParserOptions, boolean, NULL, NULL, boolean)
#22 /srv/mediawiki/php-1.36.0-wmf.11/includes/api/ApiMain.php(1565): ApiParse->execute()
#23 /srv/mediawiki/php-1.36.0-wmf.11/includes/api/ApiMain.php(545): ApiMain->executeAction()
#24 /srv/mediawiki/php-1.36.0-wmf.11/includes/api/ApiMain.php(516): ApiMain->executeActionWithErrorHandling()
#25 /srv/mediawiki/php-1.36.0-wmf.11/api.php(90): ApiMain->execute()
#26 /srv/mediawiki/php-1.36.0-wmf.11/api.php(45): wfApiMain()
#27 /srv/mediawiki/w/api.php(3): require(string)
#28 {main}

Event Timeline

mmodell created this task.Wed, Sep 30, 9:02 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptWed, Sep 30, 9:02 PM
mmodell triaged this task as Unbreak Now! priority.Wed, Sep 30, 9:09 PM
cscott added a subscriber: cscott.Wed, Sep 30, 9:14 PM

Hm, strange, codesearch doesn't know anything about LathMathML...
https://codesearch.wmcloud.org/search/?q=LathMathML&i=nope&files=&repos=
Must be a constructed class name somehow?

This is a bug in the Math extension's ParserAfterTidy hook:

#0 /srv/mediawiki/php-1.36.0-wmf.11/includes/HookContainer/HookContainer.php(333): MathHooks::onParserAfterTidy(Parser, string)
cscott added a subscriber: DannyS712.EditedWed, Sep 30, 9:16 PM

Maybe 3f16b9f1c2af13dfc4e9758e17fa9d086f9c35a4 or caa51dc4c9bd2b3dcbc7516d5f24d7531b37f1f1 (@DannyS712 )? Those were the most recent changes to MathHooks, and both took place since wmf.10....

Hm, strange, codesearch doesn't know anything about LathMathML...
https://codesearch.wmcloud.org/search/?q=LathMathML&i=nope&files=&repos=
Must be a constructed class name somehow?

Not only that, but it doesn't look like anything with the extension has changed that would cause this

rEMATcaa51dc4c9bd: Remove fallback to $wgUser in MathHooks::onPageRenderingHash - my commit
rEMATdb617781c137: build: Updating mediawiki/mediawiki-phan-config to 0.10.3 - bot commit
rEMAT3f16b9f1c2af: Remove use of $wgUser in onMaintenanceRefreshLinksInit - my commit
rEMAT18b8b1f29e03: Localisation updates from https://translatewiki.net. - bot commit

I double checked my patches, but they didn't touch anything like this

This might be a corruption bug. There's a call to MathMathML::batchEvaluate on line 352 of MathHooks::onParserAfterTidy. Seems like PHP is corrupting its strings somehow to turn MathMathML into LathMathML, and then crashing?

One of the log traces is https://logstash.wikimedia.org/app/kibana#/doc/logstash-*/logstash-deploy-2020.09.30/mediawiki?id=AXTg4LY_LNRtRo5XnEue&_g=h@44136fa which is line 356 of wmf-10.

Line number 356 of /srv/mediawiki/php-1.36.0-wmf.10/extensions/Math/src/MathHooks.php:356 is MathMathML::batchEvaluate( self::$tags );

cscott added a comment.EditedWed, Sep 30, 9:26 PM

@ssastry noticed that the crash is reported on line 356. That's the line with MathMathML::batchEvaluate *in wmf.10*. @DannyS712's patches changed the line numbers just enough so that the line with MathMathML::batchEvaluate is now line 352.

So this is definitely pointing to some deployment issue: we're crashing on a line which *in wmf.10* contained a MathMathML which "at the time of the crash" seems to contain LathMathML but which *in wmf.11 is a comment*.

So, that does lend credence to @cscott's theory of some corruption here during deploy. Are the PHP bytecode caches cleared out properly?

It does seem like something was massively wrong because the number of crazy and unrelated errors were off the chart, and the error rate didn't return to normal until I rolled back even group0 to wmf.10

I had about 20 bots running and all crashed with errors like:

ERROR: Detected MediaWiki API exception internal_api_error_Error: [ba1d3188-6194-41dc-8306-24e1f42c3c02] Caught exception of type Error

In logstash I found : [ba1d3188-6194-41dc-8306-24e1f42c3c02] /w/api.php Error from line 356 of /srv/mediawiki/php-1.36.0-wmf.10/extensions/Math/src/MathHooks.php: Class 'LathMathML' not found

Search also returns some Wikibase errors that don't seem to be related?

I'm going to untag the parsing-team and parser since I don't think this is related to both of those tags. Tagging serviceops in case they have insight into this.

CDanis added a subscriber: CDanis.

This is opcache corruption.

It occurred on just a small number of machines, and restarting php-fpm on them cleared it.