Page MenuHomePhabricator
Create Task
Maniphest T264241

Class 'LathMathML' not found
Closed, DuplicatePublicPRODUCTION ERROR
Actions

Description

Error

MediaWiki version: 1.36.0-wmf.11

message
Class 'LathMathML' not found

Impact

train blocker, large volume of errors in production

Notes

Details

Request ID
920b444c-ae68-4573-a4cf-61cd5572479c
Request URL
https://commons.wikimedia.org/w/api.php
Stack Trace
exception.trace
#0 /srv/mediawiki/php-1.36.0-wmf.11/includes/HookContainer/HookContainer.php(333): MathHooks::onParserAfterTidy(Parser, string)
#1 /srv/mediawiki/php-1.36.0-wmf.11/includes/HookContainer/HookContainer.php(140): MediaWiki\HookContainer\HookContainer->callLegacyHook(string, array, array, array)
#2 /srv/mediawiki/php-1.36.0-wmf.11/includes/HookContainer/HookRunner.php(2879): MediaWiki\HookContainer\HookContainer->run(string, array)
#3 /srv/mediawiki/php-1.36.0-wmf.11/includes/parser/Parser.php(1680): MediaWiki\HookContainer\HookRunner->onParserAfterTidy(Parser, string)
#4 /srv/mediawiki/php-1.36.0-wmf.11/includes/parser/Parser.php(649): Parser->internalParseHalfParsed(string, boolean, boolean)
#5 /srv/mediawiki/php-1.36.0-wmf.11/includes/content/WikitextContent.php(374): Parser->parse(string, Title, ParserOptions, boolean, boolean, integer)
#6 /srv/mediawiki/php-1.36.0-wmf.11/includes/content/AbstractContent.php(590): WikitextContent->fillParserOutput(Title, integer, ParserOptions, boolean, ParserOutput)
#7 /srv/mediawiki/php-1.36.0-wmf.11/includes/Revision/RenderedRevision.php(266): AbstractContent->getParserOutput(Title, integer, ParserOptions, boolean)
#8 /srv/mediawiki/php-1.36.0-wmf.11/includes/Revision/RenderedRevision.php(235): MediaWiki\Revision\RenderedRevision->getSlotParserOutputUncached(WikitextContent, boolean)
#9 /srv/mediawiki/php-1.36.0-wmf.11/includes/Revision/RevisionRenderer.php(230): MediaWiki\Revision\RenderedRevision->getSlotParserOutput(string, array)
#10 /srv/mediawiki/php-1.36.0-wmf.11/includes/Revision/RevisionRenderer.php(152): MediaWiki\Revision\RevisionRenderer->combineSlotOutput(MediaWiki\Revision\RenderedRevision, array)
#11 [internal function]: MediaWiki\Revision\RevisionRenderer->MediaWiki\Revision\{closure}(MediaWiki\Revision\RenderedRevision, array)
#12 /srv/mediawiki/php-1.36.0-wmf.11/includes/Revision/RenderedRevision.php(197): call_user_func(Closure, MediaWiki\Revision\RenderedRevision, array)
#13 /srv/mediawiki/php-1.36.0-wmf.11/includes/poolcounter/PoolWorkArticleView.php(216): MediaWiki\Revision\RenderedRevision->getRevisionParserOutput()
#14 /srv/mediawiki/php-1.36.0-wmf.11/includes/poolcounter/PoolCounterWork.php(162): PoolWorkArticleView->doWork()
#15 /srv/mediawiki/php-1.36.0-wmf.11/includes/page/WikiPage.php(1221): PoolCounterWork->execute()
#16 /srv/mediawiki/php-1.36.0-wmf.11/includes/api/ApiParse.php(83): WikiPage->getParserOutput(ParserOptions, integer, boolean)
#17 /srv/mediawiki/php-1.36.0-wmf.11/includes/poolcounter/PoolCounterWorkViaCallback.php(74): ApiParse->{closure}()
#18 /srv/mediawiki/php-1.36.0-wmf.11/includes/poolcounter/PoolCounterWork.php(162): PoolCounterWorkViaCallback->doWork()
#19 /srv/mediawiki/php-1.36.0-wmf.11/includes/api/ApiParse.php(90): PoolCounterWork->execute()
#20 /srv/mediawiki/php-1.36.0-wmf.11/includes/api/ApiParse.php(658): ApiParse->getPageParserOutput(WikiFilePage, NULL, ParserOptions, boolean)
#21 /srv/mediawiki/php-1.36.0-wmf.11/includes/api/ApiParse.php(222): ApiParse->getParsedContent(WikiFilePage, ParserOptions, boolean, NULL, NULL, boolean)
#22 /srv/mediawiki/php-1.36.0-wmf.11/includes/api/ApiMain.php(1565): ApiParse->execute()
#23 /srv/mediawiki/php-1.36.0-wmf.11/includes/api/ApiMain.php(545): ApiMain->executeAction()
#24 /srv/mediawiki/php-1.36.0-wmf.11/includes/api/ApiMain.php(516): ApiMain->executeActionWithErrorHandling()
#25 /srv/mediawiki/php-1.36.0-wmf.11/api.php(90): ApiMain->execute()
#26 /srv/mediawiki/php-1.36.0-wmf.11/api.php(45): wfApiMain()
#27 /srv/mediawiki/w/api.php(3): require(string)
#28 {main}

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolvedReleasedduvallT263177 1.36.0-wmf.11 deployment blockers
 DuplicatePRODUCTION ERRORNoneT264241 Class 'LathMathML' not found

Event Timeline

• mmodell created this task.Sep 30 2020, 9:02 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSep 30 2020, 9:02 PM
• mmodell triaged this task as Unbreak Now! priority.Sep 30 2020, 9:09 PM
• mmodell added a parent task: T263177: 1.36.0-wmf.11 deployment blockers.
cscott subscribed.Sep 30 2020, 9:14 PM

Hm, strange, codesearch doesn't know anything about LathMathML...
https://codesearch.wmcloud.org/search/?q=LathMathML&i=nope&files=&repos=
Must be a constructed class name somehow?

cscott added a comment.Sep 30 2020, 9:15 PM

This is a bug in the Math extension's ParserAfterTidy hook:

#0 /srv/mediawiki/php-1.36.0-wmf.11/includes/HookContainer/HookContainer.php(333): MathHooks::onParserAfterTidy(Parser, string)
cscott added a subscriber: DannyS712.EditedSep 30 2020, 9:16 PM

Maybe 3f16b9f1c2af13dfc4e9758e17fa9d086f9c35a4 or caa51dc4c9bd2b3dcbc7516d5f24d7531b37f1f1 (@DannyS712 )? Those were the most recent changes to MathHooks, and both took place since wmf.10....

DannyS712 added a comment.Sep 30 2020, 9:17 PM
In T264241#6506938, @cscott wrote:

Hm, strange, codesearch doesn't know anything about LathMathML...
https://codesearch.wmcloud.org/search/?q=LathMathML&i=nope&files=&repos=
Must be a constructed class name somehow?

Not only that, but it doesn't look like anything with the extension has changed that would cause this

rEMATcaa51dc4c9bd: Remove fallback to $wgUser in MathHooks::onPageRenderingHash - my commit
rEMATdb617781c137: build: Updating mediawiki/mediawiki-phan-config to 0.10.3 - bot commit
rEMAT3f16b9f1c2af: Remove use of $wgUser in onMaintenanceRefreshLinksInit - my commit
rEMAT18b8b1f29e03: Localisation updates from https://translatewiki.net. - bot commit

I double checked my patches, but they didn't touch anything like this

cscott added a comment.Sep 30 2020, 9:20 PM

This might be a corruption bug. There's a call to MathMathML::batchEvaluate on line 352 of MathHooks::onParserAfterTidy. Seems like PHP is corrupting its strings somehow to turn MathMathML into LathMathML, and then crashing?

ssastry subscribed.Sep 30 2020, 9:24 PM

One of the log traces is https://logstash.wikimedia.org/app/kibana#/doc/logstash-*/logstash-deploy-2020.09.30/mediawiki?id=AXTg4LY_LNRtRo5XnEue&_g=h@44136fa which is line 356 of wmf-10.

Line number 356 of /srv/mediawiki/php-1.36.0-wmf.10/extensions/Math/src/MathHooks.php:356 is MathMathML::batchEvaluate( self::$tags );

cscott added a comment.EditedSep 30 2020, 9:26 PM

@ssastry noticed that the crash is reported on line 356. That's the line with MathMathML::batchEvaluate *in wmf.10*. @DannyS712's patches changed the line numbers just enough so that the line with MathMathML::batchEvaluate is now line 352.

So this is definitely pointing to some deployment issue: we're crashing on a line which *in wmf.10* contained a MathMathML which "at the time of the crash" seems to contain LathMathML but which *in wmf.11 is a comment*.

ssastry added a comment.Sep 30 2020, 9:26 PM

So, that does lend credence to @cscott's theory of some corruption here during deploy. Are the PHP bytecode caches cleared out properly?

• mmodell added a comment.Sep 30 2020, 9:29 PM

It does seem like something was massively wrong because the number of crazy and unrelated errors were off the chart, and the error rate didn't return to normal until I rolled back even group0 to wmf.10

Multichill subscribed.Sep 30 2020, 9:40 PM

I had about 20 bots running and all crashed with errors like:

ERROR: Detected MediaWiki API exception internal_api_error_Error: [ba1d3188-6194-41dc-8306-24e1f42c3c02] Caught exception of type Error

In logstash I found : [ba1d3188-6194-41dc-8306-24e1f42c3c02] /w/api.php Error from line 356 of /srv/mediawiki/php-1.36.0-wmf.10/extensions/Math/src/MathHooks.php: Class 'LathMathML' not found

Search also returns some Wikibase errors that don't seem to be related?

ssastry edited projects, added serviceops; removed Parsing-Team--ARCHIVED, MediaWiki-Parser.Sep 30 2020, 10:05 PM

I'm going to untag the parsing-team and parser since I don't think this is related to both of those tags. Tagging serviceops in case they have insight into this.

CDanis closed this task as a duplicate of T245183: PHP7 corruption reports in 2020-2022 (Call on wrong object, etc.).EditedSep 30 2020, 10:06 PM
CDanis subscribed.

This is opcache corruption.

It occurred on just a small number of machines, and restarting php-fpm on them cleared it.

• Lucas_Werkmeister_WMDE mentioned this in T264243: Argument 2 passed to Wikibase\Repo\Store\TypeDispatchingEntityTitleStoreLookup::__construct() must be an instance of Wikibase\Repo\Store\EntityTitleStoreLookup, instance of Wikibase\Repo\Content\EntityContentFactory given.Oct 1 2020, 10:54 AM
ssastry mentioned this in T245183: PHP7 corruption reports in 2020-2022 (Call on wrong object, etc.).Oct 1 2020, 4:52 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL