Page MenuHomePhabricator
Create Task
Maniphest T301309

Refreshlinks job is parsing pages twice
Closed, ResolvedPublic
Actions

Description

The code jumps through lots of hoops to avoid double parsing (e.g. by locking the page, etc.) but it actually parses the page twice inside the job:
https://logstash.wikimedia.org/app/discover#/doc/logstash-*/logstash-mediawiki-2022.02.09?id=sWJW3H4BoAyk87sq2wWx

Details

SubjectRepoBranchLines +/-
RefreshLinksJob: Generate HTML when parsing if DerivedPageDataUpdater will need it latermediawiki/coremaster+21 -1
DerivedPageDataUpdater: Set ParserOutput when it's passed to itmediawiki/corewmf/1.38.0-wmf.21+3 -0
DerivedPageDataUpdater: Set ParserOutput when it's passed to itmediawiki/corewmf/1.38.0-wmf.20+3 -0
DerivedPageDataUpdater: Set ParserOutput when it's passed to itmediawiki/coremaster+3 -0
Customize query in gerrit

Related Objects
Search...

Event Timeline

Ladsgroup created this task.Feb 9 2022, 2:43 AM
gerritbot added a comment.Feb 9 2022, 10:42 PM

Change 761487 had a related patch set uploaded (by Ladsgroup; author: Amir Sarabadani):

[mediawiki/core@master] DerivedPageDataUpdater: Set ParserOutput when it's passed to it

https://gerrit.wikimedia.org/r/761487

gerritbot added a project: Patch-For-Review.Feb 9 2022, 10:42 PM
gerritbot added a comment.Feb 10 2022, 4:39 AM

Change 761487 merged by jenkins-bot:

[mediawiki/core@master] DerivedPageDataUpdater: Set ParserOutput when it's passed to it

https://gerrit.wikimedia.org/r/761487

ReleaseTaggerBot added a project: MW-1.38-notes (1.38.0-wmf.22; 2022-02-14).Feb 10 2022, 5:00 AM
Maintenance_bot removed a project: Patch-For-Review.Feb 10 2022, 5:10 AM
Krinkle edited projects, added MediaWiki-Page-derived-data; removed MediaWiki-Parser.Feb 10 2022, 5:13 AM
gerritbot added a comment.Feb 10 2022, 11:32 AM

Change 761413 had a related patch set uploaded (by Ladsgroup; author: Amir Sarabadani):

[mediawiki/core@wmf/1.38.0-wmf.20] DerivedPageDataUpdater: Set ParserOutput when it's passed to it

https://gerrit.wikimedia.org/r/761413

gerritbot added a project: Patch-For-Review.Feb 10 2022, 11:32 AM

Change 761414 had a related patch set uploaded (by Ladsgroup; author: Amir Sarabadani):

[mediawiki/core@wmf/1.38.0-wmf.21] DerivedPageDataUpdater: Set ParserOutput when it's passed to it

https://gerrit.wikimedia.org/r/761414

gerritbot added a comment.Feb 10 2022, 3:11 PM

Change 761413 merged by jenkins-bot:

[mediawiki/core@wmf/1.38.0-wmf.20] DerivedPageDataUpdater: Set ParserOutput when it's passed to it

https://gerrit.wikimedia.org/r/761413

gerritbot added a comment.Feb 10 2022, 3:11 PM

Change 761414 merged by jenkins-bot:

[mediawiki/core@wmf/1.38.0-wmf.21] DerivedPageDataUpdater: Set ParserOutput when it's passed to it

https://gerrit.wikimedia.org/r/761414

Stashbot added a comment.Feb 10 2022, 3:32 PM

Mentioned in SAL (#wikimedia-operations) [2022-02-10T15:32:38Z] <ladsgroup@deploy1002> Synchronized php-1.38.0-wmf.21/includes/Storage/DerivedPageDataUpdater.php: Backport: [[gerrit:761414|DerivedPageDataUpdater: Set ParserOutput when it's passed to it (T301309)]] (duration: 00m 53s)

Stashbot added a comment.Feb 10 2022, 3:39 PM

Mentioned in SAL (#wikimedia-operations) [2022-02-10T15:39:24Z] <ladsgroup@deploy1002> Synchronized php-1.38.0-wmf.20/includes/Storage/DerivedPageDataUpdater.php: Backport: [[gerrit:761413|DerivedPageDataUpdater: Set ParserOutput when it's passed to it (T301309)]] (duration: 00m 50s)

ReleaseTaggerBot edited projects, added MW-1.38-notes (1.38.0-wmf.21; 2022-02-07); removed MW-1.38-notes (1.38.0-wmf.22; 2022-02-14).Feb 10 2022, 4:00 PM
Maintenance_bot removed a project: Patch-For-Review.Feb 10 2022, 4:10 PM
Ladsgroup claimed this task.Feb 10 2022, 4:15 PM
Ladsgroup added a project: DBA.

https://logstash.wikimedia.org/goto/d321bfa7cf35ad3b7121bfcbff7f995a

image.png (202×1 px, 27 KB)

Restricted Application added a project: User-Ladsgroup. · View Herald TranscriptFeb 10 2022, 4:15 PM
Ladsgroup closed this task as Resolved.Feb 10 2022, 4:15 PM
Ladsgroup moved this task from Triage to Done on the DBA board.
Maintenance_bot moved this task from Incoming to Done on the User-Ladsgroup board.Feb 10 2022, 5:15 PM
matmarex subscribed.Aug 10 2023, 9:06 PM

This either hasn't worked or has regressed, because we're seeing duplicate parses inside RefreshLinksJob today. Example: https://logstash.wikimedia.org/app/discover#/doc/logstash-*/logstash-mediawiki-1-7.0.0-1-2023.08.10?id=F4s54YkBRv1FAtdn4GRO

#0 /srv/mediawiki/php-1.41.0-wmf.20/includes/content/ContentHandler.php(1790): MediaWiki\Parser\ParserObserver->notifyParse(Object(MediaWiki\Title\Title), 740306188, Object(ParserOptions), Object(Wikibase\MediaInfo\Content\MediaInfoContent), Object(ParserOutput))
#1 /srv/mediawiki/php-1.41.0-wmf.20/includes/content/Renderer/ContentRenderer.php(47): ContentHandler->getParserOutput(Object(Wikibase\MediaInfo\Content\MediaInfoContent), Object(MediaWiki\Content\Renderer\ContentParseParams))
#2 /srv/mediawiki/php-1.41.0-wmf.20/includes/Revision/RenderedRevision.php(260): MediaWiki\Content\Renderer\ContentRenderer->getParserOutput(Object(Wikibase\MediaInfo\Content\MediaInfoContent), Object(MediaWiki\Title\Title), 740306188, Object(ParserOptions), false)
#3 /srv/mediawiki/php-1.41.0-wmf.20/includes/Revision/RenderedRevision.php(232): MediaWiki\Revision\RenderedRevision->getSlotParserOutputUncached(Object(Wikibase\MediaInfo\Content\MediaInfoContent), false)
#4 /srv/mediawiki/php-1.41.0-wmf.20/includes/Revision/RevisionRenderer.php(242): MediaWiki\Revision\RenderedRevision->getSlotParserOutput('mediainfo', Array)
#5 /srv/mediawiki/php-1.41.0-wmf.20/includes/Revision/RevisionRenderer.php(164): MediaWiki\Revision\RevisionRenderer->combineSlotOutput(Object(MediaWiki\Revision\RenderedRevision), Array)
#6 [internal function]: MediaWiki\Revision\RevisionRenderer->MediaWiki\Revision\{closure}(Object(MediaWiki\Revision\RenderedRevision), Array)
#7 /srv/mediawiki/php-1.41.0-wmf.20/includes/Revision/RenderedRevision.php(199): call_user_func(Object(Closure), Object(MediaWiki\Revision\RenderedRevision), Array)
#8 /srv/mediawiki/php-1.41.0-wmf.20/includes/jobqueue/jobs/RefreshLinksJob.php(330): MediaWiki\Revision\RenderedRevision->getRevisionParserOutput(Array)
#9 /srv/mediawiki/php-1.41.0-wmf.20/includes/jobqueue/jobs/RefreshLinksJob.php(230): RefreshLinksJob->getParserOutput(Object(MediaWiki\Revision\RevisionRenderer), Object(ParserCache), Object(WikiFilePage), Object(BufferingStatsdDataFactory))
#10 /srv/mediawiki/php-1.41.0-wmf.20/includes/jobqueue/jobs/RefreshLinksJob.php(162): RefreshLinksJob->runForTitle(Object(MediaWiki\Title\Title))
#11 /srv/mediawiki/php-1.41.0-wmf.20/extensions/EventBus/includes/JobExecutor.php(78): RefreshLinksJob->run()
#12 /srv/mediawiki/rpc/RunSingleJob.php(77): MediaWiki\Extension\EventBus\JobExecutor->execute(Array)
#13 {main}
#0 /srv/mediawiki/php-1.41.0-wmf.20/includes/content/ContentHandler.php(1790): MediaWiki\Parser\ParserObserver->notifyParse(Object(MediaWiki\Title\Title), 740306188, Object(ParserOptions), Object(Wikibase\MediaInfo\Content\MediaInfoContent), Object(ParserOutput))
#1 /srv/mediawiki/php-1.41.0-wmf.20/includes/content/Renderer/ContentRenderer.php(47): ContentHandler->getParserOutput(Object(Wikibase\MediaInfo\Content\MediaInfoContent), Object(MediaWiki\Content\Renderer\ContentParseParams))
#2 /srv/mediawiki/php-1.41.0-wmf.20/includes/Revision/RenderedRevision.php(260): MediaWiki\Content\Renderer\ContentRenderer->getParserOutput(Object(Wikibase\MediaInfo\Content\MediaInfoContent), Object(MediaWiki\Title\Title), 740306188, Object(ParserOptions), true)
#3 /srv/mediawiki/php-1.41.0-wmf.20/includes/Revision/RenderedRevision.php(232): MediaWiki\Revision\RenderedRevision->getSlotParserOutputUncached(Object(Wikibase\MediaInfo\Content\MediaInfoContent), true)
#4 /srv/mediawiki/php-1.41.0-wmf.20/includes/Revision/RevisionRenderer.php(242): MediaWiki\Revision\RenderedRevision->getSlotParserOutput('mediainfo', Array)
#5 /srv/mediawiki/php-1.41.0-wmf.20/includes/Revision/RevisionRenderer.php(164): MediaWiki\Revision\RevisionRenderer->combineSlotOutput(Object(MediaWiki\Revision\RenderedRevision), Array)
#6 [internal function]: MediaWiki\Revision\RevisionRenderer->MediaWiki\Revision\{closure}(Object(MediaWiki\Revision\RenderedRevision), Array)
#7 /srv/mediawiki/php-1.41.0-wmf.20/includes/Revision/RenderedRevision.php(199): call_user_func(Object(Closure), Object(MediaWiki\Revision\RenderedRevision), Array)
#8 /srv/mediawiki/php-1.41.0-wmf.20/includes/Storage/DerivedPageDataUpdater.php(1472): MediaWiki\Revision\RenderedRevision->getRevisionParserOutput()
#9 /srv/mediawiki/php-1.41.0-wmf.20/includes/Storage/DerivedPageDataUpdater.php(1504): MediaWiki\Storage\DerivedPageDataUpdater->getCanonicalParserOutput()
#10 /srv/mediawiki/php-1.41.0-wmf.20/includes/deferred/RefreshSecondaryDataUpdate.php(85): MediaWiki\Storage\DerivedPageDataUpdater->getSecondaryDataUpdates(false)
#11 /srv/mediawiki/php-1.41.0-wmf.20/includes/deferred/DeferredUpdatesManager.php(506): RefreshSecondaryDataUpdate->doUpdate()
#12 /srv/mediawiki/php-1.41.0-wmf.20/includes/deferred/DeferredUpdates.php(277): MediaWiki\Deferred\DeferredUpdatesManager->attemptUpdate(Object(RefreshSecondaryDataUpdate), Object(Wikimedia\Rdbms\LBFactoryMulti))
#13 /srv/mediawiki/php-1.41.0-wmf.20/includes/Storage/DerivedPageDataUpdater.php(1836): DeferredUpdates::attemptUpdate(Object(RefreshSecondaryDataUpdate), Object(Wikimedia\Rdbms\LBFactoryMulti))
#14 /srv/mediawiki/php-1.41.0-wmf.20/includes/page/WikiPage.php(2100): MediaWiki\Storage\DerivedPageDataUpdater->doSecondaryDataUpdates(Array)
#15 /srv/mediawiki/php-1.41.0-wmf.20/includes/jobqueue/jobs/RefreshLinksJob.php(244): WikiPage->doSecondaryDataUpdates(Array)
#16 /srv/mediawiki/php-1.41.0-wmf.20/includes/jobqueue/jobs/RefreshLinksJob.php(162): RefreshLinksJob->runForTitle(Object(MediaWiki\Title\Title))
#17 /srv/mediawiki/php-1.41.0-wmf.20/extensions/EventBus/includes/JobExecutor.php(78): RefreshLinksJob->run()
#18 /srv/mediawiki/rpc/RunSingleJob.php(77): MediaWiki\Extension\EventBus\JobExecutor->execute(Array)
#19 {main}
matmarex added a comment.Aug 10 2023, 9:18 PM

It looks like the first parse is done with 'generate-html' => false, but the second is with true, so we have to parse again.

matmarex reopened this task as Open.Aug 10 2023, 9:59 PM
matmarex claimed this task.
gerritbot added a comment.Aug 10 2023, 10:01 PM

Change 947959 had a related patch set uploaded (by Bartosz Dziewoński; author: Bartosz Dziewoński):

[mediawiki/core@master] RefreshLinksJob: Generate HTML when parsing if DerivedPageDataUpdater will need it later

https://gerrit.wikimedia.org/r/947959

gerritbot added a project: Patch-For-Review.Aug 10 2023, 10:01 PM
Krinkle edited projects, added MediaWiki-Platform-Team; removed MW-1.38-notes (1.38.0-wmf.21; 2022-02-07), Performance-Team (Radar), Platform Team Workboards (MW Expedition).Aug 10 2023, 10:13 PM
Krinkle moved this task from Inbox, needs triage to Current Sprint on the MediaWiki-Platform-Team board.Aug 10 2023, 10:24 PM
gerritbot added a comment.Aug 18 2023, 8:07 AM

Change 947959 merged by jenkins-bot:

[mediawiki/core@master] RefreshLinksJob: Generate HTML when parsing if DerivedPageDataUpdater will need it later

https://gerrit.wikimedia.org/r/947959

Maintenance_bot removed a project: Patch-For-Review.Aug 18 2023, 8:10 AM
matmarex closed this task as Resolved.Aug 24 2023, 7:53 PM

Seems to have worked: https://logstash.wikimedia.org/goto/e68e6d825f4870401ebffdde0311d781

image.png (411×1 px, 44 KB)

ReleaseTaggerBot added a project: MW-1.41-notes (1.41.0-wmf.25; 2023-09-05).Aug 29 2023, 8:01 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits