Page MenuHomePhabricator

A revision of https://wikidata.beta.wmflabs.org/wiki/Q64 consistently doesn't load on beta wikidata
Open, MediumPublicBUG REPORT

Description

https://wikidata.beta.wmflabs.org/w/index.php?title=Q64&oldid=1161819 consitantly doesn't load on beta wikidata
It looks like this might be due to the number of terms in the revision

I get a mixture of:

Request from XXX via deployment-cache-text05 frontend, Varnish XID 2378651
Upstream caches: deployment-cache-text05 int
Error: 500, Internal Server Error at Tue, 31 Mar 2020 16:45:42 GMT
PHP fatal error:
Allowed memory size of 692060160 bytes exhausted (tried to allocate 8388608 bytes)

Event Timeline

Addshore created this task.Mar 31 2020, 4:45 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 31 2020, 4:45 PM
DannyS712 added a subscriber: DannyS712.EditedMar 31 2020, 4:47 PM

Can reproduce - PHP fatal error: Allowed memory size of 692060160 bytes exhausted (tried to allocate 69632 bytes)

DannyS712 added a comment.EditedMar 31 2020, 4:50 PM

I reverted to a prior version, and now it can be accessed, so the original issue is resolved

But, if you look at https://wikidata.beta.wmflabs.org/w/index.php?title=Q64&action=history and attempt to revert my edit and add back the 15,136‎ bytes, it fails for the same reason of memory allocation:
PHP fatal error: Allowed memory size of 692060160 bytes exhausted (tried to allocate 1052672 bytes)

Addshore added a comment.EditedMar 31 2020, 4:50 PM

Interesting.
Let me try to reproduce on a different item.

I reverted to a prior version, and now it can be accessed, so the original issue is resolved

But, if you look at https://wikidata.beta.wmflabs.org/w/index.php?title=Q64&action=history and attempt to revert my edit and add back the 15,136‎ bytes, it fails for the same reason of memory allocation:
PHP fatal error: Allowed memory size of 692060160 bytes exhausted (tried to allocate 1052672 bytes)

You also cannot view the old versions

No difference in the attempted memory for the different revisions?

Addshore added a comment.EditedMar 31 2020, 5:03 PM

Something has obviously changed, and it relates to the number of terms that have been added to the item in https://wikidata.beta.wmflabs.org/w/index.php?title=Q64&oldid=1161819
Viewing these revisions obviously used to work as the item has been editing many times since the creation of that revision without problem.
I wonder if this is a core of Wikibase change, probably something relating to lots of Language objects.
This could possibly end up impacting production, perhaps

DannyS712 triaged this task as High priority.Mar 31 2020, 5:05 PM
DannyS712 added a project: User-DannyS712.
DannyS712 changed the subtype of this task from "Task" to "Bug Report".
DannyS712 moved this task from Unsorted to Others on the User-DannyS712 board.
DannyS712 added a project: Regression.

Is anything showing up on logstash that may be of use?

I also get errors on beta while trying to use wbeditentity to make an edit adding lots of terms.
I have prepared test wikidata with a revision that breaks beta currently https://test.wikidata.org/w/index.php?title=Q64&action=history so we can see what happens as the train rolls forward.
Now going to look at recent code changes to see if I spot anything.

Addshore renamed this task from https://wikidata.beta.wmflabs.org/wiki/Q64 consitantly doesn't load on beta wikidata to A revision of https://wikidata.beta.wmflabs.org/wiki/Q64 consitantly doesn't load on beta wikidata.Apr 1 2020, 8:11 AM
Addshore updated the task description. (Show Details)
DannyS712 renamed this task from A revision of https://wikidata.beta.wmflabs.org/wiki/Q64 consitantly doesn't load on beta wikidata to A revision of https://wikidata.beta.wmflabs.org/wiki/Q64 consistently doesn't load on beta wikidata.Apr 1 2020, 8:13 AM

So, diffs actually load fine that relate to the revision. but the revision can not render (or seemingly save).

When rendering the stacktrace for the memory allocation issue looks like:

#0 /srv/mediawiki/php-master/includes/cache/localisation/LCStoreStaticArray.php(136): require()
#1 /srv/mediawiki/php-master/includes/cache/localisation/LocalisationCache.php(386): LCStoreStaticArray->get()
#2 /srv/mediawiki/php-master/includes/cache/localisation/LocalisationCache.php(304): LocalisationCache->loadItem()
#3 /srv/mediawiki/php-master/includes/cache/localisation/LocalisationCache.php(500): LocalisationCache->getItem()
#4 /srv/mediawiki/php-master/includes/cache/localisation/LocalisationCache.php(363): LocalisationCache->initLanguage()
#5 /srv/mediawiki/php-master/includes/cache/localisation/LocalisationCache.php(304): LocalisationCache->loadItem()
#6 /srv/mediawiki/php-master/includes/language/LanguageFallback.php(106): LocalisationCache->getItem()
#7 /srv/mediawiki/php-master/includes/language/LanguageFactory.php(167): MediaWiki\Languages\LanguageFallback->getAll()
#8 /srv/mediawiki/php-master/includes/language/LanguageFactory.php(114): MediaWiki\Languages\LanguageFactory->newFromCode()
#9 /srv/mediawiki/php-master/languages/Language.php(327): MediaWiki\Languages\LanguageFactory->getLanguage()
#10 /srv/mediawiki/php-master/extensions/Wikibase/repo/includes/MediaWikiLanguageDirectionalityLookup.php(27): Language::factory()
#11 /srv/mediawiki/php-master/extensions/Wikibase/view/src/TermsListView.php(144): Wikibase\Repo\MediaWikiLanguageDirectionalityLookup->getDirectionality()
#12 /srv/mediawiki/php-master/extensions/Wikibase/view/src/TermsListView.php(119): Wikibase\View\TermsListView->getTermView()
#13 /srv/mediawiki/php-master/extensions/Wikibase/repo/includes/ParserOutput/PlaceholderEmittingEntityTermsView.php(132): Wikibase\View\TermsListView->getListItemHtml()
#14 /srv/mediawiki/php-master/extensions/Wikibase/repo/includes/ParserOutput/PlaceholderEmittingEntityTermsView.php(156): Wikibase\Repo\ParserOutput\PlaceholderEmittingEntityTermsView->getTermsListItems()
#15 /srv/mediawiki/php-master/extensions/Wikibase/view/src/ItemView.php(105): Wikibase\Repo\ParserOutput\PlaceholderEmittingEntityTermsView->getPlaceholders()
#16 /srv/mediawiki/php-master/extensions/Wikibase/repo/includes/ParserOutput/FullEntityParserOutputGenerator.php(233): Wikibase\View\ItemView->getContent()
#17 /srv/mediawiki/php-master/extensions/Wikibase/repo/includes/ParserOutput/FullEntityParserOutputGenerator.php(161): Wikibase\Repo\ParserOutput\FullEntityParserOutputGenerator->addHtmlToParserOutput()
#18 /srv/mediawiki/php-master/extensions/Wikibase/repo/includes/ParserOutput/StatsdTimeRecordingEntityParserOutputGenerator.php(48): Wikibase\Repo\ParserOutput\FullEntityParserOutputGenerator->getParserOutput()
#19 /srv/mediawiki/php-master/extensions/Wikibase/repo/includes/Content/EntityContent.php(242): Wikibase\Repo\ParserOutput\StatsdTimeRecordingEntityParserOutputGenerator->getParserOutput()
#20 /srv/mediawiki/php-master/extensions/Wikibase/repo/includes/Content/ItemContent.php(234): Wikibase\EntityContent->getParserOutputFromEntityView()
#21 /srv/mediawiki/php-master/extensions/Wikibase/repo/includes/Content/EntityContent.php(181): Wikibase\ItemContent->getParserOutputFromEntityView()
#22 /srv/mediawiki/php-master/includes/Revision/RenderedRevision.php(267): Wikibase\EntityContent->getParserOutput()
#23 /srv/mediawiki/php-master/includes/Revision/RenderedRevision.php(236): MediaWiki\Revision\RenderedRevision->getSlotParserOutputUncached()
#24 /srv/mediawiki/php-master/includes/Revision/RevisionRenderer.php(215): MediaWiki\Revision\RenderedRevision->getSlotParserOutput()
#25 /srv/mediawiki/php-master/includes/Revision/RevisionRenderer.php(152): MediaWiki\Revision\RevisionRenderer->combineSlotOutput()
#26 [internal function]: MediaWiki\Revision\RevisionRenderer->MediaWiki\Revision\{closure}()
#27 /srv/mediawiki/php-master/includes/Revision/RenderedRevision.php(198): call_user_func()
#28 /srv/mediawiki/php-master/includes/poolcounter/PoolWorkArticleView.php(196): MediaWiki\Revision\RenderedRevision->getRevisionParserOutput()
#29 /srv/mediawiki/php-master/includes/poolcounter/PoolCounterWork.php(125): PoolWorkArticleView->doWork()
#30 /srv/mediawiki/php-master/includes/page/Article.php(787): PoolCounterWork->execute()
#31 /srv/mediawiki/php-master/extensions/Wikibase/repo/includes/Actions/ViewEntityAction.php(79): Article->view()
#32 /srv/mediawiki/php-master/extensions/Wikibase/repo/includes/Actions/ViewEntityAction.php(54): Wikibase\ViewEntityAction->showEntityPage()
#33 /srv/mediawiki/php-master/includes/MediaWiki.php(519): Wikibase\ViewEntityAction->show()
#34 /srv/mediawiki/php-master/includes/MediaWiki.php(305): MediaWiki->performAction()
#35 /srv/mediawiki/php-master/includes/MediaWiki.php(973): MediaWiki->performRequest()
#36 /srv/mediawiki/php-master/includes/MediaWiki.php(535): MediaWiki->main()
#37 /srv/mediawiki/php-master/index.php(47): MediaWiki->run()
#38 /srv/mediawiki/w/index.php(3): require()

Apparently when undoing / editing we fail in a similar sort of place...

#0 /srv/mediawiki/php-master/includes/cache/localisation/LCStoreStaticArray.php(136): require()
#1 /srv/mediawiki/php-master/includes/cache/localisation/LocalisationCache.php(386): LCStoreStaticArray->get()
#2 /srv/mediawiki/php-master/includes/cache/localisation/LocalisationCache.php(304): LocalisationCache->loadItem()
#3 /srv/mediawiki/php-master/includes/cache/localisation/LocalisationCache.php(500): LocalisationCache->getItem()
#4 /srv/mediawiki/php-master/includes/cache/localisation/LocalisationCache.php(363): LocalisationCache->initLanguage()
#5 /srv/mediawiki/php-master/includes/cache/localisation/LocalisationCache.php(304): LocalisationCache->loadItem()
#6 /srv/mediawiki/php-master/includes/language/LanguageFallback.php(106): LocalisationCache->getItem()
#7 /srv/mediawiki/php-master/includes/language/LanguageFactory.php(167): MediaWiki\Languages\LanguageFallback->getAll()
#8 /srv/mediawiki/php-master/includes/language/LanguageFactory.php(114): MediaWiki\Languages\LanguageFactory->newFromCode()
#9 /srv/mediawiki/php-master/languages/Language.php(327): MediaWiki\Languages\LanguageFactory->getLanguage()
#10 /srv/mediawiki/php-master/extensions/Wikibase/repo/includes/MediaWikiLanguageDirectionalityLookup.php(27): Language::factory()
#11 /srv/mediawiki/php-master/extensions/Wikibase/view/src/TermsListView.php(144): Wikibase\Repo\MediaWikiLanguageDirectionalityLookup->getDirectionality()
#12 /srv/mediawiki/php-master/extensions/Wikibase/view/src/TermsListView.php(119): Wikibase\View\TermsListView->getTermView()
#13 /srv/mediawiki/php-master/extensions/Wikibase/repo/includes/ParserOutput/PlaceholderEmittingEntityTermsView.php(132): Wikibase\View\TermsListView->getListItemHtml()
#14 /srv/mediawiki/php-master/extensions/Wikibase/repo/includes/ParserOutput/PlaceholderEmittingEntityTermsView.php(156): Wikibase\Repo\ParserOutput\PlaceholderEmittingEntityTermsView->getTermsListItems()
#15 /srv/mediawiki/php-master/extensions/Wikibase/view/src/ItemView.php(105): Wikibase\Repo\ParserOutput\PlaceholderEmittingEntityTermsView->getPlaceholders()
#16 /srv/mediawiki/php-master/extensions/Wikibase/repo/includes/ParserOutput/FullEntityParserOutputGenerator.php(233): Wikibase\View\ItemView->getContent()
#17 /srv/mediawiki/php-master/extensions/Wikibase/repo/includes/ParserOutput/FullEntityParserOutputGenerator.php(161): Wikibase\Repo\ParserOutput\FullEntityParserOutputGenerator->addHtmlToParserOutput()
#18 /srv/mediawiki/php-master/extensions/Wikibase/repo/includes/ParserOutput/StatsdTimeRecordingEntityParserOutputGenerator.php(48): Wikibase\Repo\ParserOutput\FullEntityParserOutputGenerator->getParserOutput()
#19 /srv/mediawiki/php-master/extensions/Wikibase/repo/includes/Content/EntityContent.php(242): Wikibase\Repo\ParserOutput\StatsdTimeRecordingEntityParserOutputGenerator->getParserOutput()
#20 /srv/mediawiki/php-master/extensions/Wikibase/repo/includes/Content/ItemContent.php(234): Wikibase\EntityContent->getParserOutputFromEntityView()
#21 /srv/mediawiki/php-master/extensions/Wikibase/repo/includes/Content/EntityContent.php(181): Wikibase\ItemContent->getParserOutputFromEntityView()
#22 /srv/mediawiki/php-master/includes/Revision/RenderedRevision.php(267): Wikibase\EntityContent->getParserOutput()
#23 /srv/mediawiki/php-master/includes/Revision/RenderedRevision.php(236): MediaWiki\Revision\RenderedRevision->getSlotParserOutputUncached()
#24 /srv/mediawiki/php-master/includes/Revision/RevisionRenderer.php(215): MediaWiki\Revision\RenderedRevision->getSlotParserOutput()
#25 /srv/mediawiki/php-master/includes/Revision/RevisionRenderer.php(152): MediaWiki\Revision\RevisionRenderer->combineSlotOutput()
#26 [internal function]: MediaWiki\Revision\RevisionRenderer->MediaWiki\Revision\{closure}()
#27 /srv/mediawiki/php-master/includes/Revision/RenderedRevision.php(198): call_user_func()
#28 /srv/mediawiki/php-master/includes/Storage/DerivedPageDataUpdater.php(1301): MediaWiki\Revision\RenderedRevision->getRevisionParserOutput()
#29 /srv/mediawiki/php-master/includes/Storage/PageUpdater.php(732): MediaWiki\Storage\DerivedPageDataUpdater->getCanonicalParserOutput()
#30 /srv/mediawiki/php-master/includes/page/WikiPage.php(1927): MediaWiki\Storage\PageUpdater->saveRevision()
#31 /srv/mediawiki/php-master/extensions/Wikibase/repo/includes/Actions/SubmitEntityAction.php(226): WikiPage->doEditContent()
#32 /srv/mediawiki/php-master/extensions/Wikibase/repo/includes/Actions/SubmitEntityAction.php(132): Wikibase\SubmitEntityAction->attemptSave()
#33 /srv/mediawiki/php-master/extensions/Wikibase/repo/includes/Actions/SubmitEntityAction.php(69): Wikibase\SubmitEntityAction->undo()
#34 /srv/mediawiki/php-master/includes/MediaWiki.php(519): Wikibase\SubmitEntityAction->show()
#35 /srv/mediawiki/php-master/includes/MediaWiki.php(305): MediaWiki->performAction()
#36 /srv/mediawiki/php-master/includes/MediaWiki.php(973): MediaWiki->performRequest()
#37 /srv/mediawiki/php-master/includes/MediaWiki.php(535): MediaWiki->main()
#38 /srv/mediawiki/php-master/index.php(47): MediaWiki->run()
#39 /srv/mediawiki/w/index.php(3): require()
Addshore claimed this task.Apr 1 2020, 8:31 AM
Addshore added a project: Wikidata-Campsite.
Restricted Application added a project: User-Addshore. · View Herald TranscriptApr 1 2020, 8:31 AM

Moving to the iteration so I keep my eye on it and investigate throughout this train rollout..

Maintenance_bot moved this task from incoming to in progress on the Wikidata board.Apr 1 2020, 9:15 AM
Addshore lowered the priority of this task from High to Medium.Apr 4 2020, 9:18 AM

It doesn't look like this rolled out with the train or impacted production.
Still need to investigate this a bit more to see if the situation can be improved on beta though.
Probably needs some profiling.

Addshore removed Addshore as the assignee of this task.Apr 4 2020, 9:18 AM

Now just gives Service Temporarily Unavailable Our servers are currently under maintenance or experiencing a technical problem. Please try again in a few minutes. on the beta cluster with no more information

This comment was removed by RhinosF1.

@DannyS712: loads fine here

tried again and got Allowed memory size of 692060160 bytes exhausted (tried to allocate 4194304 bytes)

I know what I did. Sorry, still half alseep.

If you report this error to the Wikimedia System Administrators, please include the details below.

PHP fatal error:
Allowed memory size of 692060160 bytes exhausted (tried to allocate 1179648 bytes)

I can't reproduce this anymore - can this be closed?