Page MenuHomePhabricator

Job unable to create file page "Fatal: Call function inNamespace() on null" (via MinervaHooks)
Closed, ResolvedPublic3 Estimate Story Points

Description

Error

MediaWiki version: 1.35.0-wmf.19

message
Fatal error: Call to a member function inNamespace() on null
exception.trace
#0 /srv/mediawiki/php-1.35.0-wmf.19/skins/MinervaNeue/includes/MinervaHooks.php(299): MinervaHooks::setMinervaSkinOptions(MobileContext, SkinMinerva)
#1 /srv/mediawiki/php-1.35.0-wmf.19/includes/Hooks.php(174): MinervaHooks::onRequestContextCreateSkinMobile(MobileContext, SkinMinerva)
#2 /srv/mediawiki/php-1.35.0-wmf.19/includes/Hooks.php(202): Hooks::callHook(string, array, array, NULL)
#3 /srv/mediawiki/php-1.35.0-wmf.19/extensions/MobileFrontend/includes/MobileFrontendHooks.php(128): Hooks::run(string, array)
#4 /srv/mediawiki/php-1.35.0-wmf.19/includes/Hooks.php(174): MobileFrontendHooks::onRequestContextCreateSkin(RequestContext, SkinMinerva)
#5 /srv/mediawiki/php-1.35.0-wmf.19/includes/Hooks.php(202): Hooks::callHook(string, array, array, NULL)
#6 /srv/mediawiki/php-1.35.0-wmf.19/includes/context/RequestContext.php(394): Hooks::run(string, array)
#7 /srv/mediawiki/php-1.35.0-wmf.19/includes/parser/ParserOutput.php(344): RequestContext->getSkin()
#8 /srv/mediawiki/php-1.35.0-wmf.19/extensions/CommonsMetadata/src/DataCollector.php(122): ParserOutput->getText()
#9 /srv/mediawiki/php-1.35.0-wmf.19/extensions/CommonsMetadata/src/HookHandler.php(145): CommonsMetadata\DataCollector->verifyAttributionMetadata(ParserOutput, LocalFile)
#10 /srv/mediawiki/php-1.35.0-wmf.19/includes/Hooks.php(174): CommonsMetadata\HookHandler::onContentAlterParserOutput(WikitextContent, Title, ParserOutput)
#11 /srv/mediawiki/php-1.35.0-wmf.19/includes/Hooks.php(202): Hooks::callHook(string, array, array, NULL)
#12 /srv/mediawiki/php-1.35.0-wmf.19/includes/content/AbstractContent.php(569): Hooks::run(string, array)
#13 /srv/mediawiki/php-1.35.0-wmf.19/includes/Revision/RenderedRevision.php(267): AbstractContent->getParserOutput(Title, NULL, ParserOptions, boolean)
#14 /srv/mediawiki/php-1.35.0-wmf.19/includes/Revision/RenderedRevision.php(236): MediaWiki\Revision\RenderedRevision->getSlotParserOutputUncached(WikitextContent, boolean)
#15 /srv/mediawiki/php-1.35.0-wmf.19/includes/Revision/RevisionRenderer.php(215): MediaWiki\Revision\RenderedRevision->getSlotParserOutput(string)
#16 /srv/mediawiki/php-1.35.0-wmf.19/includes/Revision/RevisionRenderer.php(152): MediaWiki\Revision\RevisionRenderer->combineSlotOutput(MediaWiki\Revision\RenderedRevision, array)
#17 [internal function]: MediaWiki\Revision\RevisionRenderer->MediaWiki\Revision\{closure}(MediaWiki\Revision\RenderedRevision, array)
#18 /srv/mediawiki/php-1.35.0-wmf.19/includes/Revision/RenderedRevision.php(198): call_user_func(Closure, MediaWiki\Revision\RenderedRevision, array)
#19 /srv/mediawiki/php-1.35.0-wmf.19/includes/Storage/DerivedPageDataUpdater.php(1302): MediaWiki\Revision\RenderedRevision->getRevisionParserOutput()
#20 /srv/mediawiki/php-1.35.0-wmf.19/includes/Storage/PageUpdater.php(735): MediaWiki\Storage\DerivedPageDataUpdater->getCanonicalParserOutput()
#21 /srv/mediawiki/php-1.35.0-wmf.19/includes/page/WikiPage.php(1942): MediaWiki\Storage\PageUpdater->saveRevision(CommentStoreComment, integer)
#22 /srv/mediawiki/php-1.35.0-wmf.19/includes/filerepo/file/LocalFile.php(1673): WikiPage->doEditContent(WikitextContent, CommentStoreComment, integer, boolean, User)
#23 /srv/mediawiki/php-1.35.0-wmf.19/includes/deferred/AutoCommitUpdate.php(46): LocalFile->{closure}(Wikimedia\Rdbms\MaintainableDBConnRef, string)
#24 /srv/mediawiki/php-1.35.0-wmf.19/includes/deferred/DeferredUpdates.php(420): AutoCommitUpdate->doUpdate()
#25 /srv/mediawiki/php-1.35.0-wmf.19/includes/deferred/DeferredUpdates.php(296): DeferredUpdates::attemptUpdate(AutoCommitUpdate, Wikimedia\Rdbms\LBFactoryMulti)
#26 /srv/mediawiki/php-1.35.0-wmf.19/includes/deferred/DeferredUpdates.php(233): DeferredUpdates::run(AutoCommitUpdate, Wikimedia\Rdbms\LBFactoryMulti, Monolog\Logger, BufferingStatsdDataFactory, string)
#27 /srv/mediawiki/php-1.35.0-wmf.19/includes/deferred/DeferredUpdates.php(146): DeferredUpdates::handleUpdateQueue(array, string, integer)
#28 /srv/mediawiki/php-1.35.0-wmf.19/extensions/EventBus/includes/JobExecutor.php(97): DeferredUpdates::doUpdates()
#29 /srv/mediawiki/rpc/RunSingleJob.php(76): JobExecutor->execute(array)
#30 {main}

Impact

This is fatally by aborting would-be edits to wiki pages. The specific example above appears to be (failing to) create the File description page for an image uploaded to Commons.

The upload queued a PublishStashedFileJob job, which then stores the media file and the above is meant to create the file description page.

From what I can tell, there is no recovery from this. It looks like the user input is permanently lost, which is unfortunate because it's processed from a job which means the user is no longer there to see it and try again.

Notes

  • First stack:
    • User upload -> … job queue … -> PublishStashedFileJob -> UploadBase::performUpload -> … -> LocalFile::recordUpload2 -> WikiPage::doEditContent
  • Second stack:
    • WikiPage::doEditContent -> … parse wikitext … -> Hooks onContentAlterParserOutput
  • Third stack:
    • onContentAlterParserOutput hook
    • CommonsMetadata\HookHandler::onContentAlterParserOutput
    • ParserOutput->getText
    • RequestContent->getSkin <= BOOM
    • MobileFrontendHooks::onRequestContextCreateSkin
    • MinervaHooks::onRequestContextCreateSkinMobile
    • null->inNamespace()

Details

Request ID
XkVHNQpAIDQAAEOABQ4AAABN
Related Gerrit Patches:
mediawiki/skins/MinervaNeue : wmf/1.35.0-wmf.19Check title value before proceeding to check if user page
mediawiki/skins/MinervaNeue : wmf/1.35.0-wmf.20Check title value before proceeding to check if user page
mediawiki/skins/MinervaNeue : masterCheck title value before proceeding to check if user page

Event Timeline

Krinkle created this task.Thu, Feb 13, 3:36 PM
Restricted Application added subscribers: Masumrezarock100, Aklapper. · View Herald TranscriptThu, Feb 13, 3:36 PM
Krinkle renamed this task from Fatal: Call function inNamespace() on null (from MinervaHooks) to Job unable to create file page "Fatal: Call function inNamespace() on null" (via MinervaHooks).Thu, Feb 13, 3:36 PM
Krinkle added a subscriber: brion.

@brion This might explain something you were noticing about uploads not succeeding in some cases?

Krinkle added a subscriber: Cparle.

@Cparle Not due to WikibaseMediaInfo I think, but can have cascading effects, so FYI.

Restricted Application added a project: Structured-Data-Backlog. · View Herald TranscriptThu, Feb 13, 3:38 PM
Krinkle triaged this task as Unbreak Now! priority.EditedThu, Feb 13, 3:38 PM

Marking unbreak now as this is causing data loss within a job that can't be retried afaik.

Perhaps caused by the recent changes to ParserOutput::getText, which uses RequestContent which Minerva expects to to be fully-skinned and titled, which isn't the case when called from session-less contexts such as load.php, rest.php, maintenance scripts and jobs.

More samples at https://logstash.wikimedia.org/goto/0b200b20f5bfbbf5537cd2f74834b7d9. Looks it affected about ~ 135 pages in the past month.

Restricted Application added a subscriber: Liuxinyu970226. · View Herald TranscriptThu, Feb 13, 3:38 PM

@matmarex, sorry if this doesn't make sense but do you think there's a chance https://gerrit.wikimedia.org/r/#/c/mediawiki/core/+/541597/ could be contributing to this issue?

@matmarex, sorry if this doesn't make sense but do you think there's a chance https://gerrit.wikimedia.org/r/#/c/mediawiki/core/+/541597/ could be contributing to this issue?

That code went out first in wmf.18; this bug is new in wmf.19, then it might contribute but not be the whole fix?

Krinkle added a comment.EditedThu, Feb 13, 4:31 PM

While concerning nonetheless, it does not appear to be new this week. I just haven't triaged for week.

@Niedzielski Yeah that patch might be the trigger. Although it isn't the source. Minerva's code is using $context->getTitle() from a hook that isn't limited to page views, which means it must call hasTitle() first and have a way of handling the "else" case.

Separate from this, the fact that ParserOutput::getText uses RequestContext likely means that very many hooks and extension points that were thought to only ever happen when we're in a pageview context now also happen outside of it. ParserOutput is quite a low level and commonly interacted with object for background/internal tasks. Decoupling that would certainly help. See also my code review suggestion (link) which would involve making it default to someting like new SkinFallback. However that means any code using ParserOutput from a place that does relate to page views, will no longer magically involve the "current" skin for doEditSectionLink logic. Any places that need that must use the skin option instead as the inline @todo foretold. I don't know how many or few call sites there are in that category. Might be an easy fix?

This issue doesn't seem to be related to the code I touched in https://gerrit.wikimedia.org/r/c/mediawiki/core/+/541597 and https://gerrit.wikimedia.org/r/c/mediawiki/skins/MinervaNeue/+/541598. Those parts of Minerva handle null title correctly, as far as I can see (and there's even a comment stating that "Title may be undefined in certain contexts (T179833)").

FYI there is bug T179884: Files occasionally getting uploaded to Commons without file pages., where people sometimes post new cases of file uploads without description pages, some of which might be caused by this.

Let me know if you need my help in resolving this, for now I'm assuming Reading team is handling it.

Jdlrobson added a subscriber: Jdlrobson.

This is unbreak now since yesterday and sounds like we are on the hook so I'm moving to the kanbanana board for further investigation.

Change 572728 had a related patch set uploaded (by Jdlrobson; owner: Jdlrobson):
[mediawiki/skins/MinervaNeue@master] Check hasTitle before proceeding to check if user page

https://gerrit.wikimedia.org/r/572728

Jdlrobson added a comment.EditedMon, Feb 17, 6:38 PM

I'm unable to replicate this exact workflow.

Without really understanding what's going on here (I imagine what's happening is RequestContextCreateSkin hook is running outside a page view - although I'm not sure what the workflow is for that).

Based on what I do know - that inNamespace should not be called on a Title that does not exist - I think https://gerrit.wikimedia.org/r/572728 should take care of this.

In the Notes section I've described the callstack we saw in production. It's triggered from the job queue. The sample we saw used PublishStashedFileJob , which is queued by MW if you have local file uploads enabled and use chunked uploads. Although any job that involves wikitext parsing can cause it, e.g. template updates.

It happens because ParserOutput::getText uses a Skin object for the enableSectionEditLinks feature. Which is currently grabbing it directly from RequestContext, instead of requiring it as parameter and/or having a deterministic fallback that doesn't involve the global context.

ovasileva set the point value for this task to 3.Tue, Feb 18, 5:26 PM

Change 572728 merged by jenkins-bot:
[mediawiki/skins/MinervaNeue@master] Check title value before proceeding to check if user page

https://gerrit.wikimedia.org/r/572728

Change 572991 had a related patch set uploaded (by Jdlrobson; owner: Jdlrobson):
[mediawiki/skins/MinervaNeue@wmf/1.35.0-wmf.19] Check title value before proceeding to check if user page

https://gerrit.wikimedia.org/r/572991

Anyone able to help me SWAT this today? I can't make the 4pm window. https://logstash.wikimedia.org/goto/a624d20343842ebd6f174bd7f6b842c3

Change 573346 had a related patch set uploaded (by Jdlrobson; owner: Jdlrobson):
[mediawiki/skins/MinervaNeue@wmf/1.35.0-wmf.20] Check title value before proceeding to check if user page

https://gerrit.wikimedia.org/r/573346

Change 573346 merged by jenkins-bot:
[mediawiki/skins/MinervaNeue@wmf/1.35.0-wmf.20] Check title value before proceeding to check if user page

https://gerrit.wikimedia.org/r/573346

Change 572991 merged by jenkins-bot:
[mediawiki/skins/MinervaNeue@wmf/1.35.0-wmf.19] Check title value before proceeding to check if user page

https://gerrit.wikimedia.org/r/572991

Mentioned in SAL (#wikimedia-operations) [2020-02-19T19:27:54Z] <jforrester@deploy1001> Synchronized php-1.35.0-wmf.20/skins/MinervaNeue/includes/MinervaHooks.php: T245162 Check title value before proceeding to check if user page (duration: 01m 04s)

Mentioned in SAL (#wikimedia-operations) [2020-02-19T19:31:21Z] <jforrester@deploy1001> Synchronized php-1.35.0-wmf.19/skins/MinervaNeue/includes/MinervaHooks.php: T245162 Check title value before proceeding to check if user page (duration: 01m 04s)

Waiting for a day or two worth of data before checking logstash and calling this resolved.

Jdlrobson lowered the priority of this task from Unbreak Now! to Medium.Wed, Feb 19, 7:52 PM
Jdlrobson closed this task as Resolved.Mon, Feb 24, 11:34 PM

No errors since 18th - https://logstash.wikimedia.org/goto/5a52c4374641e1eb021c82d74e355ffb
I think it is safe to say this is resolved.