Job unable to create file page "Fatal: Call function inNamespace() on null" (via MinervaHooks)
Closed, ResolvedPublic3 Estimated Story PointsPRODUCTION ERROR
Actions

Description

Error

MediaWiki version: 1.35.0-wmf.19

message

Fatal error: Call to a member function inNamespace() on null

exception.trace

#0 /srv/mediawiki/php-1.35.0-wmf.19/skins/MinervaNeue/includes/MinervaHooks.php(299): MinervaHooks::setMinervaSkinOptions(MobileContext, SkinMinerva)
#1 /srv/mediawiki/php-1.35.0-wmf.19/includes/Hooks.php(174): MinervaHooks::onRequestContextCreateSkinMobile(MobileContext, SkinMinerva)
#2 /srv/mediawiki/php-1.35.0-wmf.19/includes/Hooks.php(202): Hooks::callHook(string, array, array, NULL)
#3 /srv/mediawiki/php-1.35.0-wmf.19/extensions/MobileFrontend/includes/MobileFrontendHooks.php(128): Hooks::run(string, array)
#4 /srv/mediawiki/php-1.35.0-wmf.19/includes/Hooks.php(174): MobileFrontendHooks::onRequestContextCreateSkin(RequestContext, SkinMinerva)
#5 /srv/mediawiki/php-1.35.0-wmf.19/includes/Hooks.php(202): Hooks::callHook(string, array, array, NULL)
#6 /srv/mediawiki/php-1.35.0-wmf.19/includes/context/RequestContext.php(394): Hooks::run(string, array)
#7 /srv/mediawiki/php-1.35.0-wmf.19/includes/parser/ParserOutput.php(344): RequestContext->getSkin()
#8 /srv/mediawiki/php-1.35.0-wmf.19/extensions/CommonsMetadata/src/DataCollector.php(122): ParserOutput->getText()
#9 /srv/mediawiki/php-1.35.0-wmf.19/extensions/CommonsMetadata/src/HookHandler.php(145): CommonsMetadata\DataCollector->verifyAttributionMetadata(ParserOutput, LocalFile)
#10 /srv/mediawiki/php-1.35.0-wmf.19/includes/Hooks.php(174): CommonsMetadata\HookHandler::onContentAlterParserOutput(WikitextContent, Title, ParserOutput)
#11 /srv/mediawiki/php-1.35.0-wmf.19/includes/Hooks.php(202): Hooks::callHook(string, array, array, NULL)
#12 /srv/mediawiki/php-1.35.0-wmf.19/includes/content/AbstractContent.php(569): Hooks::run(string, array)
#13 /srv/mediawiki/php-1.35.0-wmf.19/includes/Revision/RenderedRevision.php(267): AbstractContent->getParserOutput(Title, NULL, ParserOptions, boolean)
#14 /srv/mediawiki/php-1.35.0-wmf.19/includes/Revision/RenderedRevision.php(236): MediaWiki\Revision\RenderedRevision->getSlotParserOutputUncached(WikitextContent, boolean)
#15 /srv/mediawiki/php-1.35.0-wmf.19/includes/Revision/RevisionRenderer.php(215): MediaWiki\Revision\RenderedRevision->getSlotParserOutput(string)
#16 /srv/mediawiki/php-1.35.0-wmf.19/includes/Revision/RevisionRenderer.php(152): MediaWiki\Revision\RevisionRenderer->combineSlotOutput(MediaWiki\Revision\RenderedRevision, array)
#17 [internal function]: MediaWiki\Revision\RevisionRenderer->MediaWiki\Revision\{closure}(MediaWiki\Revision\RenderedRevision, array)
#18 /srv/mediawiki/php-1.35.0-wmf.19/includes/Revision/RenderedRevision.php(198): call_user_func(Closure, MediaWiki\Revision\RenderedRevision, array)
#19 /srv/mediawiki/php-1.35.0-wmf.19/includes/Storage/DerivedPageDataUpdater.php(1302): MediaWiki\Revision\RenderedRevision->getRevisionParserOutput()
#20 /srv/mediawiki/php-1.35.0-wmf.19/includes/Storage/PageUpdater.php(735): MediaWiki\Storage\DerivedPageDataUpdater->getCanonicalParserOutput()
#21 /srv/mediawiki/php-1.35.0-wmf.19/includes/page/WikiPage.php(1942): MediaWiki\Storage\PageUpdater->saveRevision(CommentStoreComment, integer)
#22 /srv/mediawiki/php-1.35.0-wmf.19/includes/filerepo/file/LocalFile.php(1673): WikiPage->doEditContent(WikitextContent, CommentStoreComment, integer, boolean, User)
#23 /srv/mediawiki/php-1.35.0-wmf.19/includes/deferred/AutoCommitUpdate.php(46): LocalFile->{closure}(Wikimedia\Rdbms\MaintainableDBConnRef, string)
#24 /srv/mediawiki/php-1.35.0-wmf.19/includes/deferred/DeferredUpdates.php(420): AutoCommitUpdate->doUpdate()
#25 /srv/mediawiki/php-1.35.0-wmf.19/includes/deferred/DeferredUpdates.php(296): DeferredUpdates::attemptUpdate(AutoCommitUpdate, Wikimedia\Rdbms\LBFactoryMulti)
#26 /srv/mediawiki/php-1.35.0-wmf.19/includes/deferred/DeferredUpdates.php(233): DeferredUpdates::run(AutoCommitUpdate, Wikimedia\Rdbms\LBFactoryMulti, Monolog\Logger, BufferingStatsdDataFactory, string)
#27 /srv/mediawiki/php-1.35.0-wmf.19/includes/deferred/DeferredUpdates.php(146): DeferredUpdates::handleUpdateQueue(array, string, integer)
#28 /srv/mediawiki/php-1.35.0-wmf.19/extensions/EventBus/includes/JobExecutor.php(97): DeferredUpdates::doUpdates()
#29 /srv/mediawiki/rpc/RunSingleJob.php(76): JobExecutor->execute(array)
#30 {main}

Impact

This is fatally by aborting would-be edits to wiki pages. The specific example above appears to be (failing to) create the File description page for an image uploaded to Commons.

The upload queued a PublishStashedFileJob job, which then stores the media file and the above is meant to create the file description page.

From what I can tell, there is no recovery from this. It looks like the user input is permanently lost, which is unfortunate because it's processed from a job which means the user is no longer there to see it and try again.

Notes

First stack:
- User upload -> … job queue … -> PublishStashedFileJob -> UploadBase::performUpload -> … -> LocalFile::recordUpload2 -> WikiPage::doEditContent
Second stack:
- WikiPage::doEditContent -> … parse wikitext … -> Hooks onContentAlterParserOutput
Third stack:
- onContentAlterParserOutput hook
- CommonsMetadata\HookHandler::onContentAlterParserOutput
- ParserOutput->getText
- RequestContent->getSkin <= BOOM
- MobileFrontendHooks::onRequestContextCreateSkin
- MinervaHooks::onRequestContextCreateSkinMobile
- null->inNamespace()

Details

Request ID: XkVHNQpAIDQAAEOABQ4AAABN

Project	Branch	Lines +/-	Subject
mediawiki/skins/MinervaNeue	wmf/1.35.0-wmf.19	+22 -11	Check title value before proceeding to check if user page
mediawiki/skins/MinervaNeue	wmf/1.35.0-wmf.20	+22 -11	Check title value before proceeding to check if user page
mediawiki/skins/MinervaNeue	master	+22 -11	Check title value before proceeding to check if user page

Customize query in gerrit

Related Objects

Mentioned In: T301310: CommonsMetadata extension is triggering a duplicate parse in commons
T249668: 'LinkRenderer' service violates to not inspect request/session
T246537: Incomplete uploads in Upload Wizard
Mentioned Here: T179833: PHP Exception: Call to a member function isMainPage() on a non-object (null)
T179884: Files occasionally getting uploaded to Commons without file pages.

Event Timeline

Krinkle created this task.Feb 13 2020, 3:36 PM

Restricted Application added subscribers: Masumrezarock100, Aklapper. · View Herald TranscriptFeb 13 2020, 3:36 PM

@brion This might explain something you were noticing about uploads not succeeding in some cases?

@Cparle Not due to WikibaseMediaInfo I think, but can have cascading effects, so FYI.

Restricted Application added a project: Structured-Data-Backlog. · View Herald TranscriptFeb 13 2020, 3:38 PM

Marking unbreak now as this is causing data loss within a job that can't be retried afaik.

Perhaps caused by the recent changes to ParserOutput::getText, which uses RequestContent which Minerva expects to to be fully-skinned and titled, which isn't the case when called from session-less contexts such as load.php, rest.php, maintenance scripts and jobs.

More samples at https://logstash.wikimedia.org/goto/0b200b20f5bfbbf5537cd2f74834b7d9. Looks it affected about ~ 135 pages in the past month.

Restricted Application added a subscriber: Liuxinyu970226. · View Herald TranscriptFeb 13 2020, 3:38 PM

Thanks @Krinkle

@matmarex, sorry if this doesn't make sense but do you think there's a chance https://gerrit.wikimedia.org/r/#/c/mediawiki/core/+/541597/ could be contributing to this issue?

In T245162#5881327, @Niedzielski wrote:

@matmarex, sorry if this doesn't make sense but do you think there's a chance https://gerrit.wikimedia.org/r/#/c/mediawiki/core/+/541597/ could be contributing to this issue?

That code went out first in wmf.18; this bug is new in wmf.19, then it might contribute but not be the whole fix?

While concerning nonetheless, it does not appear to be new this week. I just haven't triaged for week.

@Niedzielski Yeah that patch might be the trigger. Although it isn't the source. Minerva's code is using $context->getTitle() from a hook that isn't limited to page views, which means it must call hasTitle() first and have a way of handling the "else" case.

Separate from this, the fact that ParserOutput::getText uses RequestContext likely means that very many hooks and extension points that were thought to only ever happen when we're in a pageview context now also happen outside of it. ParserOutput is quite a low level and commonly interacted with object for background/internal tasks. Decoupling that would certainly help. See also my code review suggestion (link) which would involve making it default to someting like new SkinFallback. However that means any code using ParserOutput from a place that does relate to page views, will no longer magically involve the "current" skin for doEditSectionLink logic. Any places that need that must use the skin option instead as the inline @todo foretold. I don't know how many or few call sites there are in that category. Might be an easy fix?

Krinkle moved this task from Untriaged to Feb2020/1.35-wmf.18 on the Wikimedia-production-error board.Feb 13 2020, 4:41 PM

This issue doesn't seem to be related to the code I touched in https://gerrit.wikimedia.org/r/c/mediawiki/core/+/541597 and https://gerrit.wikimedia.org/r/c/mediawiki/skins/MinervaNeue/+/541598. Those parts of Minerva handle null title correctly, as far as I can see (and there's even a comment stating that "Title may be undefined in certain contexts (T179833)").

FYI there is bug T179884: Files occasionally getting uploaded to Commons without file pages., where people sometimes post new cases of file uploads without description pages, some of which might be caused by this.

Let me know if you need my help in resolving this, for now I'm assuming Reading team is handling it.

This is unbreak now since yesterday and sounds like we are on the hook so I'm moving to the kanbanana board for further investigation.

Change 572728 had a related patch set uploaded (by Jdlrobson; owner: Jdlrobson):
[mediawiki/skins/MinervaNeue@master] Check hasTitle before proceeding to check if user page

https://gerrit.wikimedia.org/r/572728

gerritbot added a project: Patch-For-Review.Feb 17 2020, 6:36 PM

I'm unable to replicate this exact workflow.

Without really understanding what's going on here (I imagine what's happening is RequestContextCreateSkin hook is running outside a page view - although I'm not sure what the workflow is for that).

Based on what I do know - that inNamespace should not be called on a Title that does not exist - I think https://gerrit.wikimedia.org/r/572728 should take care of this.

Jdlrobson moved this task from Needs Analysis to Upcoming on the Readers-Web-Backlog (Kanbanana-2019-20-Q3) board.Feb 17 2020, 6:38 PM

In the Notes section I've described the callstack we saw in production. It's triggered from the job queue. The sample we saw used PublishStashedFileJob , which is queued by MW if you have local file uploads enabled and use chunked uploads. Although any job that involves wikitext parsing can cause it, e.g. template updates.

It happens because ParserOutput::getText uses a Skin object for the enableSectionEditLinks feature. Which is currently grabbing it directly from RequestContext, instead of requiring it as parameter and/or having a deterministic fallback that doesn't involve the global context.

ovasileva set the point value for this task to 3.Feb 18 2020, 5:26 PM

ovasileva moved this task from Upcoming to Needs Code Review on the Readers-Web-Backlog (Kanbanana-2019-20-Q3) board.

Change 572728 merged by jenkins-bot:
[mediawiki/skins/MinervaNeue@master] Check title value before proceeding to check if user page

https://gerrit.wikimedia.org/r/572728

Change 572991 had a related patch set uploaded (by Jdlrobson; owner: Jdlrobson):
[mediawiki/skins/MinervaNeue@wmf/1.35.0-wmf.19] Check title value before proceeding to check if user page

https://gerrit.wikimedia.org/r/572991

ReleaseTaggerBot added a project: MW-1.35-notes (1.35.0-wmf.21; 2020-02-25).Feb 18 2020, 10:01 PM

Anyone able to help me SWAT this today? I can't make the 4pm window. https://logstash.wikimedia.org/goto/a624d20343842ebd6f174bd7f6b842c3

MBinder_WMF assigned this task to Jdlrobson.Feb 19 2020, 6:10 PM

Change 573346 had a related patch set uploaded (by Jdlrobson; owner: Jdlrobson):
[mediawiki/skins/MinervaNeue@wmf/1.35.0-wmf.20] Check title value before proceeding to check if user page

https://gerrit.wikimedia.org/r/573346

Change 573346 merged by jenkins-bot:
[mediawiki/skins/MinervaNeue@wmf/1.35.0-wmf.20] Check title value before proceeding to check if user page

https://gerrit.wikimedia.org/r/573346

Change 572991 merged by jenkins-bot:
[mediawiki/skins/MinervaNeue@wmf/1.35.0-wmf.19] Check title value before proceeding to check if user page

https://gerrit.wikimedia.org/r/572991

Mentioned in SAL (#wikimedia-operations) [2020-02-19T19:27:54Z] <jforrester@deploy1001> Synchronized php-1.35.0-wmf.20/skins/MinervaNeue/includes/MinervaHooks.php: T245162 Check title value before proceeding to check if user page (duration: 01m 04s)

Mentioned in SAL (#wikimedia-operations) [2020-02-19T19:31:21Z] <jforrester@deploy1001> Synchronized php-1.35.0-wmf.19/skins/MinervaNeue/includes/MinervaHooks.php: T245162 Check title value before proceeding to check if user page (duration: 01m 04s)

Waiting for a day or two worth of data before checking logstash and calling this resolved.

Jdlrobson lowered the priority of this task from Unbreak Now! to Medium.Feb 19 2020, 7:52 PM

ReleaseTaggerBot edited projects, added MW-1.35-notes (1.35.0-wmf.19; 2020-02-11); removed MW-1.35-notes (1.35.0-wmf.21; 2020-02-25).Feb 19 2020, 8:00 PM

Maintenance_bot removed a project: Patch-For-Review.Feb 19 2020, 8:10 PM

No errors since 18th - https://logstash.wikimedia.org/goto/5a52c4374641e1eb021c82d74e355ffb
I think it is safe to say this is resolved.

• Ramsey-WMF mentioned this in T246537: Incomplete uploads in Upload Wizard.Mar 3 2020, 7:48 PM

Krinkle mentioned this in T249668: 'LinkRenderer' service violates to not inspect request/session.Apr 8 2020, 2:50 PM

Krinkle mentioned this in T301310: CommonsMetadata extension is triggering a duplicate parse in commons.Feb 9 2022, 11:54 PM

Job unable to create file page "Fatal: Call function inNamespace() on null" (via MinervaHooks)Closed, ResolvedPublic3 Estimated Story PointsPRODUCTION ERRORActions