Page MenuHomePhabricator

Special:Homepage runs out of memory
Closed, ResolvedPublic

Description

Allowed memory size of 698351616 bytes exhausted (tried to allocate 79691784 bytes)
#0 [internal function]: unknown()
#1 /srv/mediawiki/php-1.36.0-wmf.10/includes/libs/rdbms/database/DatabaseMysqli.php(46): mysqli->query()
#2 /srv/mediawiki/php-1.36.0-wmf.10/includes/libs/rdbms/database/Database.php(1376): Wikimedia\Rdbms\DatabaseMysqli->doQuery()
#3 /srv/mediawiki/php-1.36.0-wmf.10/includes/libs/rdbms/database/Database.php(1294): Wikimedia\Rdbms\Database->executeQueryAttempt()
#4 /srv/mediawiki/php-1.36.0-wmf.10/includes/libs/rdbms/database/Database.php(1223): Wikimedia\Rdbms\Database->executeQuery()
#5 /srv/mediawiki/php-1.36.0-wmf.10/includes/libs/rdbms/database/Database.php(1907): Wikimedia\Rdbms\Database->query()
#6 /srv/mediawiki/php-1.36.0-wmf.10/extensions/GrowthExperiments/includes/NewcomerTasks/TemplateFilter.php(134): Wikimedia\Rdbms\Database->select()
#7 /srv/mediawiki/php-1.36.0-wmf.10/extensions/GrowthExperiments/includes/NewcomerTasks/TemplateFilter.php(57): GrowthExperiments\NewcomerTasks\TemplateFilter->buildResultsMap()
#8 /srv/mediawiki/php-1.36.0-wmf.10/extensions/GrowthExperiments/includes/NewcomerTasks/TaskSuggester/CacheDecorator.php(75): GrowthExperiments\NewcomerTasks\TemplateFilter->filter()
#9 /srv/mediawiki/php-1.36.0-wmf.10/includes/libs/objectcache/wancache/WANObjectCache.php(1533): GrowthExperiments\NewcomerTasks\TaskSuggester\CacheDecorator->GrowthExperiments\NewcomerTasks\TaskSuggester\{closure}()
#10 /srv/mediawiki/php-1.36.0-wmf.10/includes/libs/objectcache/wancache/WANObjectCache.php(1376): WANObjectCache->fetchOrRegenerate()
#11 /srv/mediawiki/php-1.36.0-wmf.10/extensions/GrowthExperiments/includes/NewcomerTasks/TaskSuggester/CacheDecorator.php(99): WANObjectCache->getWithSetCallback()
#12 /srv/mediawiki/php-1.36.0-wmf.10/extensions/GrowthExperiments/includes/HomepageModules/SuggestedEdits.php(316): GrowthExperiments\NewcomerTasks\TaskSuggester\CacheDecorator->suggest()
#13 /srv/mediawiki/php-1.36.0-wmf.10/extensions/GrowthExperiments/includes/HomepageModules/SuggestedEdits.php(245): GrowthExperiments\HomepageModules\SuggestedEdits->getTaskSet()
#14 /srv/mediawiki/php-1.36.0-wmf.10/extensions/GrowthExperiments/includes/Specials/SpecialHomepage.php(376): GrowthExperiments\HomepageModules\SuggestedEdits->getJsData()
#15 /srv/mediawiki/php-1.36.0-wmf.10/extensions/GrowthExperiments/includes/Specials/SpecialHomepage.php(165): GrowthExperiments\Specials\SpecialHomepage->outputJsData()
#16 /srv/mediawiki/php-1.36.0-wmf.10/includes/specialpage/SpecialPage.php(600): GrowthExperiments\Specials\SpecialHomepage->execute()
#17 /srv/mediawiki/php-1.36.0-wmf.10/includes/specialpage/SpecialPageFactory.php(692): SpecialPage->run()
#18 /srv/mediawiki/php-1.36.0-wmf.10/includes/MediaWiki.php(307): MediaWiki\SpecialPage\SpecialPageFactory->executePath()
#19 /srv/mediawiki/php-1.36.0-wmf.10/includes/MediaWiki.php(940): MediaWiki->performRequest()
#20 /srv/mediawiki/php-1.36.0-wmf.10/includes/MediaWiki.php(543): MediaWiki->main()
#21 /srv/mediawiki/php-1.36.0-wmf.10/index.php(53): MediaWiki->run()
#22 /srv/mediawiki/php-1.36.0-wmf.10/index.php(46): wfIndexMain()
#23 /srv/mediawiki/w/index.php(3): require()

logstash

Something like 10-20 errors per hour (which given the usage of Special:Homepage seems fairly large), started with the last train, presumably related to rEGREde203e8d36e9: Add TemplateFilter for validating TaskSet.

Event Timeline

Tgr created this task.Mon, Sep 28, 6:00 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMon, Sep 28, 6:00 PM
Tgr claimed this task.Mon, Sep 28, 6:00 PM
Tgr added a comment.Mon, Sep 28, 6:03 PM

It seems to only happen on huwiki, hywiki and ukwiki. Not sure what to make of that.

Tgr added a comment.Mon, Sep 28, 6:35 PM

The error is not quite deterministic, but it is happening most of the time (meaning, Special:Homepage has been pretty much out of commission on those three wikis since Thursday).

Tried verbose logging, but of course the fatal breaks MediaWiki's internal logging mechanism.

Tgr added a comment.Mon, Sep 28, 6:58 PM

The immediate issue is Task::getTemplates() returning an empty array on those three wikis, which in turn causes TemplateFilter::buildQueryConds() to return an empty array, which will turn the query into a scan of the entire templatelinks table.

Change 630668 had a related patch set uploaded (by Gergő Tisza; owner: Gergő Tisza):
[mediawiki/extensions/GrowthExperiments@master] Prevent returning the full templatelinks table in TemplateFilter

https://gerrit.wikimedia.org/r/630668

Tgr added a comment.EditedMon, Sep 28, 7:35 PM

One thing that looks suspicious is that we define templates with a full prefix in NewcomerTasks.json, but then aren't really consistent about handling that. So it seems we end up with TitleValue objects with a namespace in the title text:

>>> $srv = \MediaWiki\MediaWikiServices::getInstance()
>>> $srv->get('GrowthExperimentsConfigurationLoader')->loadTaskTypes()[0]->getTemplates()[0]->getText()
=> "Sablon:Korrektúrázandó"

But it doesn't make any sense for this to only trigger the error on three wikis...

Also, removing the namespace didn't help.

Change 630672 had a related patch set uploaded (by Gergő Tisza; owner: Gergő Tisza):
[mediawiki/extensions/GrowthExperiments@master] Use all task templates insted of current ones in TemplateFilter

https://gerrit.wikimedia.org/r/630672

Change 630677 had a related patch set uploaded (by Gergő Tisza; owner: Gergő Tisza):
[mediawiki/extensions/GrowthExperiments@master] Properly handle namespaces in tasktype template configuration

https://gerrit.wikimedia.org/r/630677

Tgr added a comment.Mon, Sep 28, 9:14 PM

Looking at the other wikis, most of them indeed do not have namespace prefixes for their templates. euwiki is the only one I found that does, and it does show up in the logs (it was just small enough that I did not notice). I guess the only reason the edit above did not seem to help was caching (I did account for PageConfigurationLoader caching but not for the task suggester cache also storing task task / task type templates).

Change 630677 merged by jenkins-bot:
[mediawiki/extensions/GrowthExperiments@master] Properly handle namespaces in tasktype template configuration

https://gerrit.wikimedia.org/r/630677

Change 630420 had a related patch set uploaded (by Gergő Tisza; owner: Gergő Tisza):
[mediawiki/extensions/GrowthExperiments@wmf/1.36.0-wmf.10] Properly handle namespaces in tasktype template configuration

https://gerrit.wikimedia.org/r/630420

Change 630420 merged by jenkins-bot:
[mediawiki/extensions/GrowthExperiments@wmf/1.36.0-wmf.10] Properly handle namespaces in tasktype template configuration

https://gerrit.wikimedia.org/r/630420

Mentioned in SAL (#wikimedia-operations) [2020-09-28T23:42:56Z] <tgr@deploy1001> Synchronized php-1.36.0-wmf.10/extensions/GrowthExperiments/includes/NewcomerTasks/ConfigurationLoader/PageConfigurationLoader.php: Backport: [[gerrit:630420|Properly handle namespaces in tasktype template configuration (T264029)]] (duration: 01m 03s)

Change 630701 had a related patch set uploaded (by Gergő Tisza; owner: Gergő Tisza):
[mediawiki/extensions/GrowthExperiments@master] Add (and increment) CacheDecorator cache version

https://gerrit.wikimedia.org/r/630701

Change 630701 merged by jenkins-bot:
[mediawiki/extensions/GrowthExperiments@master] Add (and increment) CacheDecorator cache version

https://gerrit.wikimedia.org/r/630701

Change 630421 had a related patch set uploaded (by Gergő Tisza; owner: Gergő Tisza):
[mediawiki/extensions/GrowthExperiments@wmf/1.36.0-wmf.10] Add (and increment) CacheDecorator cache version

https://gerrit.wikimedia.org/r/630421

Change 630421 merged by jenkins-bot:
[mediawiki/extensions/GrowthExperiments@wmf/1.36.0-wmf.10] Add (and increment) CacheDecorator cache version

https://gerrit.wikimedia.org/r/630421

Change 630668 merged by jenkins-bot:
[mediawiki/extensions/GrowthExperiments@master] Prevent returning the full templatelinks table in TemplateFilter

https://gerrit.wikimedia.org/r/630668

Change 631235 had a related patch set uploaded (by Catrope; owner: Gergő Tisza):
[mediawiki/extensions/GrowthExperiments@wmf/1.36.0-wmf.10] Prevent returning the full templatelinks table in TemplateFilter

https://gerrit.wikimedia.org/r/631235

Change 631236 had a related patch set uploaded (by Catrope; owner: Gergő Tisza):
[mediawiki/extensions/GrowthExperiments@wmf/1.36.0-wmf.11] Prevent returning the full templatelinks table in TemplateFilter

https://gerrit.wikimedia.org/r/631236

Change 631235 merged by jenkins-bot:
[mediawiki/extensions/GrowthExperiments@wmf/1.36.0-wmf.10] Prevent returning the full templatelinks table in TemplateFilter

https://gerrit.wikimedia.org/r/631235

Change 631236 merged by jenkins-bot:
[mediawiki/extensions/GrowthExperiments@wmf/1.36.0-wmf.11] Prevent returning the full templatelinks table in TemplateFilter

https://gerrit.wikimedia.org/r/631236

Mentioned in SAL (#wikimedia-operations) [2020-10-01T12:12:58Z] <urbanecm@deploy1001> Synchronized php-1.36.0-wmf.10/extensions/GrowthExperiments/includes/NewcomerTasks/TemplateFilter.php: 500d0c70c84936bcdecdd0927bcbb9ff7265afa9: Prevent returning the full templatelinks table in TemplateFilter (T264029) (duration: 01m 00s)

Mentioned in SAL (#wikimedia-operations) [2020-10-01T12:15:23Z] <urbanecm@deploy1001> Synchronized php-1.36.0-wmf.11/extensions/GrowthExperiments/includes/NewcomerTasks/TemplateFilter.php: 500d0c70c84936bcdecdd0927bcbb9ff7265afa9: Prevent returning the full templatelinks table in TemplateFilter (T264029) (duration: 00m 59s)

Change 630672 merged by jenkins-bot:
[mediawiki/extensions/GrowthExperiments@master] Use all task templates instead of current ones in TemplateFilter

https://gerrit.wikimedia.org/r/630672

Urbanecm closed this task as Resolved.Wed, Oct 14, 7:34 PM
Urbanecm added a subscriber: Urbanecm.

All seems to be done?

Urbanecm edited subscribers, added: Urbanecm_WMF; removed: Urbanecm.Fri, Oct 23, 5:45 PM