Page MenuHomePhabricator
Create Task
Maniphest T264029

Special:Homepage runs out of memory
Closed, ResolvedPublic
Actions

Description

Allowed memory size of 698351616 bytes exhausted (tried to allocate 79691784 bytes)
#0 [internal function]: unknown()
#1 /srv/mediawiki/php-1.36.0-wmf.10/includes/libs/rdbms/database/DatabaseMysqli.php(46): mysqli->query()
#2 /srv/mediawiki/php-1.36.0-wmf.10/includes/libs/rdbms/database/Database.php(1376): Wikimedia\Rdbms\DatabaseMysqli->doQuery()
#3 /srv/mediawiki/php-1.36.0-wmf.10/includes/libs/rdbms/database/Database.php(1294): Wikimedia\Rdbms\Database->executeQueryAttempt()
#4 /srv/mediawiki/php-1.36.0-wmf.10/includes/libs/rdbms/database/Database.php(1223): Wikimedia\Rdbms\Database->executeQuery()
#5 /srv/mediawiki/php-1.36.0-wmf.10/includes/libs/rdbms/database/Database.php(1907): Wikimedia\Rdbms\Database->query()
#6 /srv/mediawiki/php-1.36.0-wmf.10/extensions/GrowthExperiments/includes/NewcomerTasks/TemplateFilter.php(134): Wikimedia\Rdbms\Database->select()
#7 /srv/mediawiki/php-1.36.0-wmf.10/extensions/GrowthExperiments/includes/NewcomerTasks/TemplateFilter.php(57): GrowthExperiments\NewcomerTasks\TemplateFilter->buildResultsMap()
#8 /srv/mediawiki/php-1.36.0-wmf.10/extensions/GrowthExperiments/includes/NewcomerTasks/TaskSuggester/CacheDecorator.php(75): GrowthExperiments\NewcomerTasks\TemplateFilter->filter()
#9 /srv/mediawiki/php-1.36.0-wmf.10/includes/libs/objectcache/wancache/WANObjectCache.php(1533): GrowthExperiments\NewcomerTasks\TaskSuggester\CacheDecorator->GrowthExperiments\NewcomerTasks\TaskSuggester\{closure}()
#10 /srv/mediawiki/php-1.36.0-wmf.10/includes/libs/objectcache/wancache/WANObjectCache.php(1376): WANObjectCache->fetchOrRegenerate()
#11 /srv/mediawiki/php-1.36.0-wmf.10/extensions/GrowthExperiments/includes/NewcomerTasks/TaskSuggester/CacheDecorator.php(99): WANObjectCache->getWithSetCallback()
#12 /srv/mediawiki/php-1.36.0-wmf.10/extensions/GrowthExperiments/includes/HomepageModules/SuggestedEdits.php(316): GrowthExperiments\NewcomerTasks\TaskSuggester\CacheDecorator->suggest()
#13 /srv/mediawiki/php-1.36.0-wmf.10/extensions/GrowthExperiments/includes/HomepageModules/SuggestedEdits.php(245): GrowthExperiments\HomepageModules\SuggestedEdits->getTaskSet()
#14 /srv/mediawiki/php-1.36.0-wmf.10/extensions/GrowthExperiments/includes/Specials/SpecialHomepage.php(376): GrowthExperiments\HomepageModules\SuggestedEdits->getJsData()
#15 /srv/mediawiki/php-1.36.0-wmf.10/extensions/GrowthExperiments/includes/Specials/SpecialHomepage.php(165): GrowthExperiments\Specials\SpecialHomepage->outputJsData()
#16 /srv/mediawiki/php-1.36.0-wmf.10/includes/specialpage/SpecialPage.php(600): GrowthExperiments\Specials\SpecialHomepage->execute()
#17 /srv/mediawiki/php-1.36.0-wmf.10/includes/specialpage/SpecialPageFactory.php(692): SpecialPage->run()
#18 /srv/mediawiki/php-1.36.0-wmf.10/includes/MediaWiki.php(307): MediaWiki\SpecialPage\SpecialPageFactory->executePath()
#19 /srv/mediawiki/php-1.36.0-wmf.10/includes/MediaWiki.php(940): MediaWiki->performRequest()
#20 /srv/mediawiki/php-1.36.0-wmf.10/includes/MediaWiki.php(543): MediaWiki->main()
#21 /srv/mediawiki/php-1.36.0-wmf.10/index.php(53): MediaWiki->run()
#22 /srv/mediawiki/php-1.36.0-wmf.10/index.php(46): wfIndexMain()
#23 /srv/mediawiki/w/index.php(3): require()

logstash

Something like 10-20 errors per hour (which given the usage of Special:Homepage seems fairly large), started with the last train, presumably related to rEGREde203e8d36e9: Add TemplateFilter for validating TaskSet.

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
Use all task templates instead of current ones in TemplateFiltermediawiki/extensions/GrowthExperimentsmaster+11 -13
Prevent returning the full templatelinks table in TemplateFiltermediawiki/extensions/GrowthExperimentswmf/1.36.0-wmf.11+5 -0
Prevent returning the full templatelinks table in TemplateFiltermediawiki/extensions/GrowthExperimentswmf/1.36.0-wmf.10+5 -0
Prevent returning the full templatelinks table in TemplateFiltermediawiki/extensions/GrowthExperimentsmaster+5 -0
Add (and increment) CacheDecorator cache versionmediawiki/extensions/GrowthExperimentswmf/1.36.0-wmf.10+3 -0
Add (and increment) CacheDecorator cache versionmediawiki/extensions/GrowthExperimentsmaster+3 -0
Properly handle namespaces in tasktype template configurationmediawiki/extensions/GrowthExperimentswmf/1.36.0-wmf.10+35 -10
Properly handle namespaces in tasktype template configurationmediawiki/extensions/GrowthExperimentsmaster+35 -10
Customize query in gerrit

Related Objects

Event Timeline

Tgr created this task.Sep 28 2020, 6:00 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSep 28 2020, 6:00 PM
Tgr claimed this task.Sep 28 2020, 6:00 PM
kostajh subscribed.Sep 28 2020, 6:02 PM
Tgr added a comment.Sep 28 2020, 6:03 PM

It seems to only happen on huwiki, hywiki and ukwiki. Not sure what to make of that.

MMiller_WMF subscribed.Sep 28 2020, 6:17 PM
Tgr added a comment.Sep 28 2020, 6:35 PM

The error is not quite deterministic, but it is happening most of the time (meaning, Special:Homepage has been pretty much out of commission on those three wikis since Thursday).

Tried verbose logging, but of course the fatal breaks MediaWiki's internal logging mechanism.

Tgr added a comment.Sep 28 2020, 6:58 PM

The immediate issue is Task::getTemplates() returning an empty array on those three wikis, which in turn causes TemplateFilter::buildQueryConds() to return an empty array, which will turn the query into a scan of the entire templatelinks table.

gerritbot added a comment.Sep 28 2020, 7:11 PM

Change 630668 had a related patch set uploaded (by Gergő Tisza; owner: Gergő Tisza):
[mediawiki/extensions/GrowthExperiments@master] Prevent returning the full templatelinks table in TemplateFilter

https://gerrit.wikimedia.org/r/630668

gerritbot added a project: Patch-For-Review.Sep 28 2020, 7:11 PM
Tgr added a comment.EditedSep 28 2020, 7:35 PM

One thing that looks suspicious is that we define templates with a full prefix in NewcomerTasks.json, but then aren't really consistent about handling that. So it seems we end up with TitleValue objects with a namespace in the title text:

>>> $srv = \MediaWiki\MediaWikiServices::getInstance()
>>> $srv->get('GrowthExperimentsConfigurationLoader')->loadTaskTypes()[0]->getTemplates()[0]->getText()
=> "Sablon:Korrektúrázandó"

But it doesn't make any sense for this to only trigger the error on three wikis...

Also, removing the namespace didn't help.

gerritbot added a comment.Sep 28 2020, 7:50 PM

Change 630672 had a related patch set uploaded (by Gergő Tisza; owner: Gergő Tisza):
[mediawiki/extensions/GrowthExperiments@master] Use all task templates insted of current ones in TemplateFilter

https://gerrit.wikimedia.org/r/630672

gerritbot added a comment.Sep 28 2020, 8:41 PM

Change 630677 had a related patch set uploaded (by Gergő Tisza; owner: Gergő Tisza):
[mediawiki/extensions/GrowthExperiments@master] Properly handle namespaces in tasktype template configuration

https://gerrit.wikimedia.org/r/630677

Tgr added a comment.Sep 28 2020, 9:14 PM

Looking at the other wikis, most of them indeed do not have namespace prefixes for their templates. euwiki is the only one I found that does, and it does show up in the logs (it was just small enough that I did not notice). I guess the only reason the edit above did not seem to help was caching (I did account for PageConfigurationLoader caching but not for the task suggester cache also storing task task / task type templates).

Tgr moved this task from Incoming to Code Review on the Growth-Team (Sprint 0 (Growth Team)) board.Sep 28 2020, 9:37 PM
gerritbot added a comment.Sep 28 2020, 10:17 PM

Change 630677 merged by jenkins-bot:
[mediawiki/extensions/GrowthExperiments@master] Properly handle namespaces in tasktype template configuration

https://gerrit.wikimedia.org/r/630677

gerritbot added a comment.Sep 28 2020, 10:45 PM

Change 630420 had a related patch set uploaded (by Gergő Tisza; owner: Gergő Tisza):
[mediawiki/extensions/GrowthExperiments@wmf/1.36.0-wmf.10] Properly handle namespaces in tasktype template configuration

https://gerrit.wikimedia.org/r/630420

ReleaseTaggerBot added a project: MW-1.36-notes (1.36.0-wmf.11; 2020-09-29).Sep 28 2020, 11:00 PM
gerritbot added a comment.Sep 28 2020, 11:32 PM

Change 630420 merged by jenkins-bot:
[mediawiki/extensions/GrowthExperiments@wmf/1.36.0-wmf.10] Properly handle namespaces in tasktype template configuration

https://gerrit.wikimedia.org/r/630420

Stashbot added a comment.Sep 28 2020, 11:42 PM

Mentioned in SAL (#wikimedia-operations) [2020-09-28T23:42:56Z] <tgr@deploy1001> Synchronized php-1.36.0-wmf.10/extensions/GrowthExperiments/includes/NewcomerTasks/ConfigurationLoader/PageConfigurationLoader.php: Backport: [[gerrit:630420|Properly handle namespaces in tasktype template configuration (T264029)]] (duration: 01m 03s)

gerritbot added a comment.Sep 28 2020, 11:48 PM

Change 630701 had a related patch set uploaded (by Gergő Tisza; owner: Gergő Tisza):
[mediawiki/extensions/GrowthExperiments@master] Add (and increment) CacheDecorator cache version

https://gerrit.wikimedia.org/r/630701

ReleaseTaggerBot edited projects, added MW-1.36-notes (1.36.0-wmf.10; 2020-09-22); removed MW-1.36-notes (1.36.0-wmf.11; 2020-09-29).Sep 29 2020, 12:02 AM
gerritbot added a comment.Sep 29 2020, 12:09 AM

Change 630701 merged by jenkins-bot:
[mediawiki/extensions/GrowthExperiments@master] Add (and increment) CacheDecorator cache version

https://gerrit.wikimedia.org/r/630701

gerritbot added a comment.Sep 29 2020, 12:15 AM

Change 630421 had a related patch set uploaded (by Gergő Tisza; owner: Gergő Tisza):
[mediawiki/extensions/GrowthExperiments@wmf/1.36.0-wmf.10] Add (and increment) CacheDecorator cache version

https://gerrit.wikimedia.org/r/630421

gerritbot added a comment.Sep 29 2020, 12:23 AM

Change 630421 merged by jenkins-bot:
[mediawiki/extensions/GrowthExperiments@wmf/1.36.0-wmf.10] Add (and increment) CacheDecorator cache version

https://gerrit.wikimedia.org/r/630421

Tgr mentioned this in T264060: Outage detection for Special:Homepage.Sep 29 2020, 12:41 AM
Tgr moved this task from Code Review to QA on the Growth-Team (Sprint 0 (Growth Team)) board.

Should be fixed. Filed T264059: Make sure GrowthExperiments task suggester cache does not store outdated configuration and T264060: Outage detection for Special:Homepage as follow-ups.

ReleaseTaggerBot edited projects, added MW-1.36-notes (1.36.0-wmf.11; 2020-09-29); removed MW-1.36-notes (1.36.0-wmf.10; 2020-09-22).Sep 29 2020, 1:00 AM
gerritbot added a comment.Sep 29 2020, 9:09 AM

Change 630668 merged by jenkins-bot:
[mediawiki/extensions/GrowthExperiments@master] Prevent returning the full templatelinks table in TemplateFilter

https://gerrit.wikimedia.org/r/630668

ReleaseTaggerBot edited projects, added MW-1.36-notes (1.36.0-wmf.12; 2020-10-05; NEVER DEPLOYED); removed MW-1.36-notes (1.36.0-wmf.11; 2020-09-29).Sep 29 2020, 10:00 AM
Tacsipacsi subscribed.Sep 29 2020, 12:43 PM
gerritbot added a comment.Sep 30 2020, 8:51 PM

Change 631235 had a related patch set uploaded (by Catrope; owner: Gergő Tisza):
[mediawiki/extensions/GrowthExperiments@wmf/1.36.0-wmf.10] Prevent returning the full templatelinks table in TemplateFilter

https://gerrit.wikimedia.org/r/631235

gerritbot added a comment.Sep 30 2020, 8:52 PM

Change 631236 had a related patch set uploaded (by Catrope; owner: Gergő Tisza):
[mediawiki/extensions/GrowthExperiments@wmf/1.36.0-wmf.11] Prevent returning the full templatelinks table in TemplateFilter

https://gerrit.wikimedia.org/r/631236

gerritbot added a comment.Oct 1 2020, 12:09 PM

Change 631235 merged by jenkins-bot:
[mediawiki/extensions/GrowthExperiments@wmf/1.36.0-wmf.10] Prevent returning the full templatelinks table in TemplateFilter

https://gerrit.wikimedia.org/r/631235

gerritbot added a comment.Oct 1 2020, 12:12 PM

Change 631236 merged by jenkins-bot:
[mediawiki/extensions/GrowthExperiments@wmf/1.36.0-wmf.11] Prevent returning the full templatelinks table in TemplateFilter

https://gerrit.wikimedia.org/r/631236

Stashbot added a comment.Oct 1 2020, 12:12 PM

Mentioned in SAL (#wikimedia-operations) [2020-10-01T12:12:58Z] <urbanecm@deploy1001> Synchronized php-1.36.0-wmf.10/extensions/GrowthExperiments/includes/NewcomerTasks/TemplateFilter.php: rEGRE500d0c70c849: Prevent returning the full templatelinks table in TemplateFilter (T264029) (duration: 01m 00s)

Stashbot added a comment.Oct 1 2020, 12:15 PM

Mentioned in SAL (#wikimedia-operations) [2020-10-01T12:15:23Z] <urbanecm@deploy1001> Synchronized php-1.36.0-wmf.11/extensions/GrowthExperiments/includes/NewcomerTasks/TemplateFilter.php: rEGRE500d0c70c849: Prevent returning the full templatelinks table in TemplateFilter (T264029) (duration: 00m 59s)

ReleaseTaggerBot edited projects, added MW-1.36-notes (1.36.0-wmf.11; 2020-09-29); removed MW-1.36-notes (1.36.0-wmf.12; 2020-10-05; NEVER DEPLOYED).Oct 1 2020, 2:00 PM
Krinkle moved this task from Untriaged to Sep 2020 on the Wikimedia-production-error board.Oct 4 2020, 11:16 PM
gerritbot added a comment.Oct 5 2020, 9:39 AM

Change 630672 merged by jenkins-bot:
[mediawiki/extensions/GrowthExperiments@master] Use all task templates instead of current ones in TemplateFilter

https://gerrit.wikimedia.org/r/630672

ReleaseTaggerBot edited projects, added MW-1.36-notes (1.36.0-wmf.12; 2020-10-05; NEVER DEPLOYED); removed MW-1.36-notes (1.36.0-wmf.11; 2020-09-29).Oct 5 2020, 10:00 AM
Urbanecm closed this task as Resolved.Oct 14 2020, 7:34 PM
Urbanecm subscribed.

All seems to be done?

Urbanecm edited subscribers, added: Urbanecm_WMF; removed: Urbanecm.Oct 23 2020, 5:45 PM
kostajh mentioned this in T267216: Slow load times for Special:Homepage on cswiki.Nov 4 2020, 2:15 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits