Page MenuHomePhabricator
Create Task
Maniphest T249018

A revision of https://wikidata.beta.wmflabs.org/wiki/Q64 consistently doesn't load on beta wikidata
Closed, ResolvedPublicBUG REPORT
Actions

Description

https://wikidata.beta.wmflabs.org/w/index.php?title=Q64&oldid=1161819 consitantly doesn't load on beta wikidata
It looks like this might be due to the number of terms in the revision

I get a mixture of:

Request from XXX via deployment-cache-text05 frontend, Varnish XID 2378651
Upstream caches: deployment-cache-text05 int
Error: 500, Internal Server Error at Tue, 31 Mar 2020 16:45:42 GMT
PHP fatal error:
Allowed memory size of 692060160 bytes exhausted (tried to allocate 8388608 bytes)

Related Objects

Event Timeline

Addshore created this task.Mar 31 2020, 4:45 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 31 2020, 4:45 PM
Addshore added projects: Beta-Cluster-Infrastructure, Wikidata.Mar 31 2020, 4:45 PM
Addshore updated the task description. (Show Details)
DannyS712 subscribed.EditedMar 31 2020, 4:47 PM

Can reproduce - PHP fatal error: Allowed memory size of 692060160 bytes exhausted (tried to allocate 69632 bytes)

DannyS712 added a comment.EditedMar 31 2020, 4:50 PM

I reverted to a prior version, and now it can be accessed, so the original issue is resolved

But, if you look at https://wikidata.beta.wmflabs.org/w/index.php?title=Q64&action=history and attempt to revert my edit and add back the 15,136‎ bytes, it fails for the same reason of memory allocation:
PHP fatal error: Allowed memory size of 692060160 bytes exhausted (tried to allocate 1052672 bytes)

Addshore added a comment.EditedMar 31 2020, 4:50 PM

Interesting.
Let me try to reproduce on a different item.

DannyS712 added a comment.Mar 31 2020, 4:56 PM
In T249018#6015614, @DannyS712 wrote:

I reverted to a prior version, and now it can be accessed, so the original issue is resolved

But, if you look at https://wikidata.beta.wmflabs.org/w/index.php?title=Q64&action=history and attempt to revert my edit and add back the 15,136‎ bytes, it fails for the same reason of memory allocation:
PHP fatal error: Allowed memory size of 692060160 bytes exhausted (tried to allocate 1052672 bytes)

You also cannot view the old versions

No difference in the attempted memory for the different revisions?

Addshore added a comment.EditedMar 31 2020, 5:03 PM

Something has obviously changed, and it relates to the number of terms that have been added to the item in https://wikidata.beta.wmflabs.org/w/index.php?title=Q64&oldid=1161819
Viewing these revisions obviously used to work as the item has been editing many times since the creation of that revision without problem.
I wonder if this is a core of Wikibase change, probably something relating to lots of Language objects.
This could possibly end up impacting production, perhaps

DannyS712 triaged this task as High priority.Mar 31 2020, 5:05 PM
DannyS712 added a project: User-DannyS712.
DannyS712 changed the subtype of this task from "Task" to "Bug Report".
DannyS712 moved this task from Unsorted to Others on the User-DannyS712 board.
DannyS712 added a project: Regression.

Is anything showing up on logstash that may be of use?

Addshore added a comment.Mar 31 2020, 6:18 PM

I also get errors on beta while trying to use wbeditentity to make an edit adding lots of terms.
I have prepared test wikidata with a revision that breaks beta currently https://test.wikidata.org/w/index.php?title=Q64&action=history so we can see what happens as the train rolls forward.
Now going to look at recent code changes to see if I spot anything.

Addshore renamed this task from https://wikidata.beta.wmflabs.org/wiki/Q64 consitantly doesn't load on beta wikidata to A revision of https://wikidata.beta.wmflabs.org/wiki/Q64 consitantly doesn't load on beta wikidata.Apr 1 2020, 8:11 AM
Addshore updated the task description. (Show Details)
DannyS712 renamed this task from A revision of https://wikidata.beta.wmflabs.org/wiki/Q64 consitantly doesn't load on beta wikidata to A revision of https://wikidata.beta.wmflabs.org/wiki/Q64 consistently doesn't load on beta wikidata.Apr 1 2020, 8:13 AM
Addshore added a comment.Apr 1 2020, 8:19 AM

So, diffs actually load fine that relate to the revision. but the revision can not render (or seemingly save).

When rendering the stacktrace for the memory allocation issue looks like:

#0 /srv/mediawiki/php-master/includes/cache/localisation/LCStoreStaticArray.php(136): require()
#1 /srv/mediawiki/php-master/includes/cache/localisation/LocalisationCache.php(386): LCStoreStaticArray->get()
#2 /srv/mediawiki/php-master/includes/cache/localisation/LocalisationCache.php(304): LocalisationCache->loadItem()
#3 /srv/mediawiki/php-master/includes/cache/localisation/LocalisationCache.php(500): LocalisationCache->getItem()
#4 /srv/mediawiki/php-master/includes/cache/localisation/LocalisationCache.php(363): LocalisationCache->initLanguage()
#5 /srv/mediawiki/php-master/includes/cache/localisation/LocalisationCache.php(304): LocalisationCache->loadItem()
#6 /srv/mediawiki/php-master/includes/language/LanguageFallback.php(106): LocalisationCache->getItem()
#7 /srv/mediawiki/php-master/includes/language/LanguageFactory.php(167): MediaWiki\Languages\LanguageFallback->getAll()
#8 /srv/mediawiki/php-master/includes/language/LanguageFactory.php(114): MediaWiki\Languages\LanguageFactory->newFromCode()
#9 /srv/mediawiki/php-master/languages/Language.php(327): MediaWiki\Languages\LanguageFactory->getLanguage()
#10 /srv/mediawiki/php-master/extensions/Wikibase/repo/includes/MediaWikiLanguageDirectionalityLookup.php(27): Language::factory()
#11 /srv/mediawiki/php-master/extensions/Wikibase/view/src/TermsListView.php(144): Wikibase\Repo\MediaWikiLanguageDirectionalityLookup->getDirectionality()
#12 /srv/mediawiki/php-master/extensions/Wikibase/view/src/TermsListView.php(119): Wikibase\View\TermsListView->getTermView()
#13 /srv/mediawiki/php-master/extensions/Wikibase/repo/includes/ParserOutput/PlaceholderEmittingEntityTermsView.php(132): Wikibase\View\TermsListView->getListItemHtml()
#14 /srv/mediawiki/php-master/extensions/Wikibase/repo/includes/ParserOutput/PlaceholderEmittingEntityTermsView.php(156): Wikibase\Repo\ParserOutput\PlaceholderEmittingEntityTermsView->getTermsListItems()
#15 /srv/mediawiki/php-master/extensions/Wikibase/view/src/ItemView.php(105): Wikibase\Repo\ParserOutput\PlaceholderEmittingEntityTermsView->getPlaceholders()
#16 /srv/mediawiki/php-master/extensions/Wikibase/repo/includes/ParserOutput/FullEntityParserOutputGenerator.php(233): Wikibase\View\ItemView->getContent()
#17 /srv/mediawiki/php-master/extensions/Wikibase/repo/includes/ParserOutput/FullEntityParserOutputGenerator.php(161): Wikibase\Repo\ParserOutput\FullEntityParserOutputGenerator->addHtmlToParserOutput()
#18 /srv/mediawiki/php-master/extensions/Wikibase/repo/includes/ParserOutput/StatsdTimeRecordingEntityParserOutputGenerator.php(48): Wikibase\Repo\ParserOutput\FullEntityParserOutputGenerator->getParserOutput()
#19 /srv/mediawiki/php-master/extensions/Wikibase/repo/includes/Content/EntityContent.php(242): Wikibase\Repo\ParserOutput\StatsdTimeRecordingEntityParserOutputGenerator->getParserOutput()
#20 /srv/mediawiki/php-master/extensions/Wikibase/repo/includes/Content/ItemContent.php(234): Wikibase\EntityContent->getParserOutputFromEntityView()
#21 /srv/mediawiki/php-master/extensions/Wikibase/repo/includes/Content/EntityContent.php(181): Wikibase\ItemContent->getParserOutputFromEntityView()
#22 /srv/mediawiki/php-master/includes/Revision/RenderedRevision.php(267): Wikibase\EntityContent->getParserOutput()
#23 /srv/mediawiki/php-master/includes/Revision/RenderedRevision.php(236): MediaWiki\Revision\RenderedRevision->getSlotParserOutputUncached()
#24 /srv/mediawiki/php-master/includes/Revision/RevisionRenderer.php(215): MediaWiki\Revision\RenderedRevision->getSlotParserOutput()
#25 /srv/mediawiki/php-master/includes/Revision/RevisionRenderer.php(152): MediaWiki\Revision\RevisionRenderer->combineSlotOutput()
#26 [internal function]: MediaWiki\Revision\RevisionRenderer->MediaWiki\Revision\{closure}()
#27 /srv/mediawiki/php-master/includes/Revision/RenderedRevision.php(198): call_user_func()
#28 /srv/mediawiki/php-master/includes/poolcounter/PoolWorkArticleView.php(196): MediaWiki\Revision\RenderedRevision->getRevisionParserOutput()
#29 /srv/mediawiki/php-master/includes/poolcounter/PoolCounterWork.php(125): PoolWorkArticleView->doWork()
#30 /srv/mediawiki/php-master/includes/page/Article.php(787): PoolCounterWork->execute()
#31 /srv/mediawiki/php-master/extensions/Wikibase/repo/includes/Actions/ViewEntityAction.php(79): Article->view()
#32 /srv/mediawiki/php-master/extensions/Wikibase/repo/includes/Actions/ViewEntityAction.php(54): Wikibase\ViewEntityAction->showEntityPage()
#33 /srv/mediawiki/php-master/includes/MediaWiki.php(519): Wikibase\ViewEntityAction->show()
#34 /srv/mediawiki/php-master/includes/MediaWiki.php(305): MediaWiki->performAction()
#35 /srv/mediawiki/php-master/includes/MediaWiki.php(973): MediaWiki->performRequest()
#36 /srv/mediawiki/php-master/includes/MediaWiki.php(535): MediaWiki->main()
#37 /srv/mediawiki/php-master/index.php(47): MediaWiki->run()
#38 /srv/mediawiki/w/index.php(3): require()

Apparently when undoing / editing we fail in a similar sort of place...

#0 /srv/mediawiki/php-master/includes/cache/localisation/LCStoreStaticArray.php(136): require()
#1 /srv/mediawiki/php-master/includes/cache/localisation/LocalisationCache.php(386): LCStoreStaticArray->get()
#2 /srv/mediawiki/php-master/includes/cache/localisation/LocalisationCache.php(304): LocalisationCache->loadItem()
#3 /srv/mediawiki/php-master/includes/cache/localisation/LocalisationCache.php(500): LocalisationCache->getItem()
#4 /srv/mediawiki/php-master/includes/cache/localisation/LocalisationCache.php(363): LocalisationCache->initLanguage()
#5 /srv/mediawiki/php-master/includes/cache/localisation/LocalisationCache.php(304): LocalisationCache->loadItem()
#6 /srv/mediawiki/php-master/includes/language/LanguageFallback.php(106): LocalisationCache->getItem()
#7 /srv/mediawiki/php-master/includes/language/LanguageFactory.php(167): MediaWiki\Languages\LanguageFallback->getAll()
#8 /srv/mediawiki/php-master/includes/language/LanguageFactory.php(114): MediaWiki\Languages\LanguageFactory->newFromCode()
#9 /srv/mediawiki/php-master/languages/Language.php(327): MediaWiki\Languages\LanguageFactory->getLanguage()
#10 /srv/mediawiki/php-master/extensions/Wikibase/repo/includes/MediaWikiLanguageDirectionalityLookup.php(27): Language::factory()
#11 /srv/mediawiki/php-master/extensions/Wikibase/view/src/TermsListView.php(144): Wikibase\Repo\MediaWikiLanguageDirectionalityLookup->getDirectionality()
#12 /srv/mediawiki/php-master/extensions/Wikibase/view/src/TermsListView.php(119): Wikibase\View\TermsListView->getTermView()
#13 /srv/mediawiki/php-master/extensions/Wikibase/repo/includes/ParserOutput/PlaceholderEmittingEntityTermsView.php(132): Wikibase\View\TermsListView->getListItemHtml()
#14 /srv/mediawiki/php-master/extensions/Wikibase/repo/includes/ParserOutput/PlaceholderEmittingEntityTermsView.php(156): Wikibase\Repo\ParserOutput\PlaceholderEmittingEntityTermsView->getTermsListItems()
#15 /srv/mediawiki/php-master/extensions/Wikibase/view/src/ItemView.php(105): Wikibase\Repo\ParserOutput\PlaceholderEmittingEntityTermsView->getPlaceholders()
#16 /srv/mediawiki/php-master/extensions/Wikibase/repo/includes/ParserOutput/FullEntityParserOutputGenerator.php(233): Wikibase\View\ItemView->getContent()
#17 /srv/mediawiki/php-master/extensions/Wikibase/repo/includes/ParserOutput/FullEntityParserOutputGenerator.php(161): Wikibase\Repo\ParserOutput\FullEntityParserOutputGenerator->addHtmlToParserOutput()
#18 /srv/mediawiki/php-master/extensions/Wikibase/repo/includes/ParserOutput/StatsdTimeRecordingEntityParserOutputGenerator.php(48): Wikibase\Repo\ParserOutput\FullEntityParserOutputGenerator->getParserOutput()
#19 /srv/mediawiki/php-master/extensions/Wikibase/repo/includes/Content/EntityContent.php(242): Wikibase\Repo\ParserOutput\StatsdTimeRecordingEntityParserOutputGenerator->getParserOutput()
#20 /srv/mediawiki/php-master/extensions/Wikibase/repo/includes/Content/ItemContent.php(234): Wikibase\EntityContent->getParserOutputFromEntityView()
#21 /srv/mediawiki/php-master/extensions/Wikibase/repo/includes/Content/EntityContent.php(181): Wikibase\ItemContent->getParserOutputFromEntityView()
#22 /srv/mediawiki/php-master/includes/Revision/RenderedRevision.php(267): Wikibase\EntityContent->getParserOutput()
#23 /srv/mediawiki/php-master/includes/Revision/RenderedRevision.php(236): MediaWiki\Revision\RenderedRevision->getSlotParserOutputUncached()
#24 /srv/mediawiki/php-master/includes/Revision/RevisionRenderer.php(215): MediaWiki\Revision\RenderedRevision->getSlotParserOutput()
#25 /srv/mediawiki/php-master/includes/Revision/RevisionRenderer.php(152): MediaWiki\Revision\RevisionRenderer->combineSlotOutput()
#26 [internal function]: MediaWiki\Revision\RevisionRenderer->MediaWiki\Revision\{closure}()
#27 /srv/mediawiki/php-master/includes/Revision/RenderedRevision.php(198): call_user_func()
#28 /srv/mediawiki/php-master/includes/Storage/DerivedPageDataUpdater.php(1301): MediaWiki\Revision\RenderedRevision->getRevisionParserOutput()
#29 /srv/mediawiki/php-master/includes/Storage/PageUpdater.php(732): MediaWiki\Storage\DerivedPageDataUpdater->getCanonicalParserOutput()
#30 /srv/mediawiki/php-master/includes/page/WikiPage.php(1927): MediaWiki\Storage\PageUpdater->saveRevision()
#31 /srv/mediawiki/php-master/extensions/Wikibase/repo/includes/Actions/SubmitEntityAction.php(226): WikiPage->doEditContent()
#32 /srv/mediawiki/php-master/extensions/Wikibase/repo/includes/Actions/SubmitEntityAction.php(132): Wikibase\SubmitEntityAction->attemptSave()
#33 /srv/mediawiki/php-master/extensions/Wikibase/repo/includes/Actions/SubmitEntityAction.php(69): Wikibase\SubmitEntityAction->undo()
#34 /srv/mediawiki/php-master/includes/MediaWiki.php(519): Wikibase\SubmitEntityAction->show()
#35 /srv/mediawiki/php-master/includes/MediaWiki.php(305): MediaWiki->performAction()
#36 /srv/mediawiki/php-master/includes/MediaWiki.php(973): MediaWiki->performRequest()
#37 /srv/mediawiki/php-master/includes/MediaWiki.php(535): MediaWiki->main()
#38 /srv/mediawiki/php-master/index.php(47): MediaWiki->run()
#39 /srv/mediawiki/w/index.php(3): require()
Addshore claimed this task.Apr 1 2020, 8:31 AM
Addshore added a project: Wikidata-Campsite.
Restricted Application added a project: User-Addshore. · View Herald TranscriptApr 1 2020, 8:31 AM
Addshore edited projects, added Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)); removed Wikidata-Campsite.Apr 1 2020, 8:32 AM

Moving to the iteration so I keep my eye on it and investigate throughout this train rollout..

Addshore moved this task from To Do (prioritised from top to bottom) to Doing on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.Apr 1 2020, 8:32 AM
Addshore moved this task from Unsorted 💣 to Active 🚁 on the User-Addshore board.Apr 1 2020, 8:43 AM
Maintenance_bot moved this task from incoming to in progress on the Wikidata board.Apr 1 2020, 9:15 AM
Addshore lowered the priority of this task from High to Medium.Apr 4 2020, 9:18 AM
Addshore edited projects, added Wikidata-Campsite; removed Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)).

It doesn't look like this rolled out with the train or impacted production.
Still need to investigate this a bit more to see if the situation can be improved on beta though.
Probably needs some profiling.

Addshore removed Addshore as the assignee of this task.Apr 4 2020, 9:18 AM
Addshore mentioned this in T99740: Use static php array files for l10n cache at WMF (instead of CDB).Apr 5 2020, 5:07 PM
• Nikerabbit subscribed.Apr 6 2020, 9:26 AM
Tarrow mentioned this in T249587: Wikidata API wbgetentities: Allowed memory size exceeded.Apr 7 2020, 8:01 AM
DannyS712 added a comment.May 4 2020, 6:34 AM

Now just gives Service Temporarily Unavailable Our servers are currently under maintenance or experiencing a technical problem. Please try again in a few minutes. on the beta cluster with no more information

RhinosF1 subscribed.May 4 2020, 6:38 AM
This comment was removed by RhinosF1.
DannyS712 added a comment.May 4 2020, 6:39 AM
In T249018#6103373, @RhinosF1 wrote:

@DannyS712: loads fine here

tried again and got Allowed memory size of 692060160 bytes exhausted (tried to allocate 4194304 bytes)

RhinosF1 added a comment.May 4 2020, 6:41 AM

I know what I did. Sorry, still half alseep.

If you report this error to the Wikimedia System Administrators, please include the details below.

PHP fatal error:
Allowed memory size of 692060160 bytes exhausted (tried to allocate 1179648 bytes)

taavi subscribed.Dec 29 2020, 5:18 PM

I can't reproduce this anymore - can this be closed?

Addshore closed this task as Resolved.Apr 15 2021, 2:28 PM
Addshore claimed this task.
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits