Page MenuHomePhabricator
Create Task
Maniphest T384254

Beta cluster log spam: MediaWiki\Extension\CommunityConfiguration\Access\MediaWikiConfigReader was unable to find GEInfoboxTemplatesTest in community configuration, returning configuration from the fallback config
Closed, ResolvedPublic
Actions

Description

Very high rate starting from today, noticed while browsing the logs for another error.

MediaWiki\Extension\CommunityConfiguration\Access\MediaWikiConfigReader was unable to find GEInfoboxTemplatesTest in community configuration, returning configuration from the fallback config

from /srv/mediawiki/php-master/extensions/CommunityConfiguration/src/Access/MediaWikiConfigReader.php(120)
#0 /srv/mediawiki/php-master/extensions/CommunityConfiguration/src/Access/MediaWikiConfigReader.php(131): MediaWiki\Extension\CommunityConfiguration\Access\MediaWikiConfigReader->getConfigByVariableName(string)
#1 /srv/mediawiki/php-master/extensions/GrowthExperiments/includes/Config/MediaWikiConfigReaderWrapper.php(46): MediaWiki\Extension\CommunityConfiguration\Access\MediaWikiConfigReader->get(string)
#2 /srv/mediawiki/php-master/extensions/GrowthExperiments/includes/HomepageHooks.php(1409): GrowthExperiments\Config\MediaWikiConfigReaderWrapper->get(string)
#3 /srv/mediawiki/php-master/includes/HookContainer/HookContainer.php(155): GrowthExperiments\HomepageHooks::onCirrusSearchAddQueryFeatures(CirrusSearch\SearchConfig, array)
#4 /srv/mediawiki/php-master/extensions/CirrusSearch/includes/CirrusSearchHookRunner.php(33): MediaWiki\HookContainer\HookContainer->run(string, array)
#5 /srv/mediawiki/php-master/extensions/CirrusSearch/includes/Parser/FullTextKeywordRegistry.php(99): CirrusSearch\CirrusSearchHookRunner->onCirrusSearchAddQueryFeatures(CirrusSearch\SearchConfig, array)
#6 /srv/mediawiki/php-master/extensions/CirrusSearch/includes/Searcher.php(308): CirrusSearch\Parser\FullTextKeywordRegistry->__construct(CirrusSearch\SearchConfig)
#7 /srv/mediawiki/php-master/extensions/CirrusSearch/includes/Searcher.php(340): CirrusSearch\Searcher->buildFullTextSearch(string)
#8 /srv/mediawiki/php-master/extensions/CirrusSearch/includes/Searcher.php(219): CirrusSearch\Searcher->searchTextInternal(string)
#9 /srv/mediawiki/php-master/extensions/CirrusSearch/includes/CirrusSearch.php(305): CirrusSearch\Searcher->search(CirrusSearch\Search\SearchQuery)
#10 /srv/mediawiki/php-master/extensions/CirrusSearch/includes/CirrusSearch.php(278): CirrusSearch\CirrusSearch->searchTextReal(CirrusSearch\Search\SearchQuery)
#11 /srv/mediawiki/php-master/includes/search/SearchEngine.php(102): CirrusSearch\CirrusSearch->doSearchText(string)
#12 /srv/mediawiki/php-master/includes/search/SearchEngine.php(194): SearchEngine->{closure}()
#13 /srv/mediawiki/php-master/includes/search/SearchEngine.php(103): SearchEngine->maybePaginate(Closure)
#14 /srv/mediawiki/php-master/extensions/GrowthExperiments/includes/NewcomerTasks/TaskSuggester/LocalSearchTaskSuggester.php(130): SearchEngine->searchText(string)
#15 /srv/mediawiki/php-master/extensions/GrowthExperiments/includes/NewcomerTasks/TaskSuggester/SearchTaskSuggester.php(196): GrowthExperiments\NewcomerTasks\TaskSuggester\LocalSearchTaskSuggester->search(GrowthExperiments\NewcomerTasks\TaskSuggester\SearchStrategy\SearchQuery, int, int, bool)
#16 /srv/mediawiki/php-master/extensions/GrowthExperiments/includes/NewcomerTasks/TaskSuggester/SearchTaskSuggester.php(92): GrowthExperiments\NewcomerTasks\TaskSuggester\SearchTaskSuggester->doSuggest(null, MediaWiki\User\User, GrowthExperiments\NewcomerTasks\Task\TaskSetFilters, int, int, array)
#17 /srv/mediawiki/php-master/extensions/GrowthExperiments/includes/NewcomerTasks/TaskSuggester/LocalSearchTaskSuggester.php(73): GrowthExperiments\NewcomerTasks\TaskSuggester\SearchTaskSuggester->suggest(MediaWiki\User\User, GrowthExperiments\NewcomerTasks\Task\TaskSetFilters, int, null, array)
#18 /srv/mediawiki/php-master/extensions/GrowthExperiments/includes/NewcomerTasks/TaskSuggester/CacheDecorator.php(80): GrowthExperiments\NewcomerTasks\TaskSuggester\LocalSearchTaskSuggester->suggest(MediaWiki\User\User, GrowthExperiments\NewcomerTasks\Task\TaskSetFilters, int, null, array)
#19 /srv/mediawiki/php-master/extensions/GrowthExperiments/includes/NewcomerTasks/TaskSuggester/QualityGateDecorator.php(82): GrowthExperiments\NewcomerTasks\TaskSuggester\CacheDecorator->suggest(MediaWiki\User\User, GrowthExperiments\NewcomerTasks\Task\TaskSetFilters, int, null, array)
#20 /srv/mediawiki/php-master/extensions/GrowthExperiments/maintenance/refreshLinkRecommendations.php(245): GrowthExperiments\NewcomerTasks\TaskSuggester\QualityGateDecorator->suggest(MediaWiki\User\User, GrowthExperiments\NewcomerTasks\Task\TaskSetFilters, int, null, array)
#21 /srv/mediawiki/php-master/extensions/GrowthExperiments/maintenance/refreshLinkRecommendations.php(140): GrowthExperiments\Maintenance\RefreshLinkRecommendations->findArticlesInTopic(string)
#22 /srv/mediawiki/php-master/maintenance/includes/MaintenanceRunner.php(695): GrowthExperiments\Maintenance\RefreshLinkRecommendations->execute()
#23 /srv/mediawiki/php-master/maintenance/run.php(51): MediaWiki\Maintenance\MaintenanceRunner->run()
#24 /srv/mediawiki/multiversion/MWScript.php(156): require_once(string)
#25 {main}

https://beta-logs.wmcloud.org/goto/e9a1972b4cf937a783efb3a09cfcc4a0

image.png (2,780×532 px, 73 KB)

Details

Related Changes in Gerrit:
SubjectAuthorRepoBranchLines +/-
Remove GEInfoboxTemplatesTestUrbanecmmediawiki/extensions/GrowthExperimentswmf/1.44.0-wmf.12+0 -8
Remove GEInfoboxTemplatesTestUrbanecmmediawiki/extensions/GrowthExperimentswmf/1.44.0-wmf.13+0 -8
Remove GEInfoboxTemplatesTestUrbanecmmediawiki/extensions/GrowthExperimentsmaster+0 -8
Customize query in gerrit

Related Objects
Search...

Event Timeline

• matmarex created this task.Jan 20 2025, 6:45 PM
Restricted Application added a project: Growth-Team. · View Herald TranscriptJan 20 2025, 6:45 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
Urbanecm_WMF claimed this task.Jan 20 2025, 8:10 PM
Urbanecm_WMF triaged this task as High priority.
Urbanecm_WMF edited projects, added: GrowthExperiments-NewcomerTasks, CommunityConfiguration-Adoption, Growth-Team (Current Sprint); removed: Growth-Team, GrowthExperiments, MediaWiki-extensions-CommunityConfiguration.
Urbanecm_WMF subscribed.

Thank you! This is exactly what I was looking for in T384184. We'll fix this soon.

Urbanecm_WMF added a comment.Jan 20 2025, 8:12 PM

FTR, this is likely to become production spam when the train moves forward.

Urbanecm_WMF added a parent task: T384184: Only use GrowthExperimentsServices::getWikiConfig to request configuration variables registered in community configuration.Jan 20 2025, 8:12 PM
Aklapper mentioned this in T382364: 1.44.0-wmf.13 deployment blockers.Jan 21 2025, 3:49 PM
• Agusbou2015 subscribed.Jan 21 2025, 4:57 PM
brennen added a project: User-brennen.Jan 22 2025, 4:42 PM
brennen subscribed.

FTR, this is likely to become production spam when the train moves forward.

FWIW, I did not see this when moving the train to group0 yesterday. Do you think it's likely to be an issue at group1 and beyond?

brennen moved this task from Backlog to Logs/Train on the User-brennen board.Jan 22 2025, 4:42 PM
Michael subscribed.Jan 22 2025, 5:06 PM
Urbanecm_WMF added a comment.Jan 22 2025, 5:14 PM
In T384254#10485169, @brennen wrote:

FTR, this is likely to become production spam when the train moves forward.

FWIW, I did not see this when moving the train to group0 yesterday.

Really? I checked in production, and I do see those events coming from group0 at a fairly high volume (CommunityConfiguration.log has 250M+ of data for today, over 48k of log entries). Note group0 equals testwiki in this case, and since the events appear to be indeed appearing, their group2 volume has a chance of knocking the logging infrastructure down. FWIW, those are not errors, but logged at the INFO level (but at a fairly high rate).

Do you think it's likely to be an issue at group1 and beyond?

I wouldn't expect this to be lot worse for group1, Growth targets Wikipedias only, and there isn't a lot of them on group1. However, unless my imagination is very vivid and I'm seeing logs that are not being generated, I wouldn't recommend proceeding with group2.

brennen raised the priority of this task from High to Unbreak Now!.Jan 22 2025, 7:27 PM

Really? I checked in production, and I do see those events coming from group0 at a fairly high volume (CommunityConfiguration.log has 250M+ of data for today, over 48k of log entries). Note group0 equals testwiki in this case, and since the events appear to be indeed appearing, their group2 volume has a chance of knocking the logging infrastructure down. FWIW, those are not errors, but logged at the INFO level (but at a fairly high rate).

Ah, I see. These don't hit the places we typically monitor for the train: mediawiki-errors and "MediaWiki New Errors ECS" dashboards and the log files consolidated by logspam-watch (exception.log, error.log).

I wouldn't expect this to be lot worse for group1, Growth targets Wikipedias only, and there isn't a lot of them on group1. However, unless my imagination is very vivid and I'm seeing logs that are not being generated, I wouldn't recommend proceeding with group2.

Ok, I'm raising priority here accordingly. Seeking some further input, but I will either:

  1. Roll to group1 but block for group2, on the theory that this will be basically status quo for this group and we're better off finding other things broken
  2. Roll back, on the theory that this is more log volume than we should treat as acceptable, since it eats into our margin for other problems
brennen added a parent task: T382364: 1.44.0-wmf.13 deployment blockers.Jan 22 2025, 7:29 PM
Michael added a comment.EditedJan 22 2025, 7:34 PM

@Urbanecm_WMF Is GEInfoboxTemplatesTest actually still in use anywhere? Maybe the easiest is to just remove all of that.

I tried code-search for GEInfoboxTemplatesTest and code-search for infoboxtest. And only GrowthExperiments shows up.

Alternatively, we could probably revert MediaWikiConfigReader: Log when falling back which should also be straight forward and basically without downsides as this only added the logging that is now causing problems.

EDIT: Then again, the description for that config is "List of templates that provide Infobox functionality. Used for manually testing changes.", so maybe we don't want to throw it out on a whim. That would mean that reverting the above change might be the right way to mitigate this train-blocker, and then we can think about what we want to do about that config.

brennen added a subscriber: jeena.Jan 22 2025, 8:03 PM

Talked over with @jeena as backup. For simplicity's sake we're just going to block here.

Jdforrester-WMF subscribed.Jan 22 2025, 8:19 PM
gerritbot added a comment.Jan 23 2025, 9:28 AM

Change #1113749 had a related patch set uploaded (by Urbanecm; author: Urbanecm):

[mediawiki/extensions/GrowthExperiments@master] Remove GEInfoboxTemplatesTest

https://gerrit.wikimedia.org/r/1113749

gerritbot added a project: Patch-For-Review.Jan 23 2025, 9:28 AM
gerritbot added a comment.Jan 23 2025, 10:14 AM

Change #1113749 merged by jenkins-bot:

[mediawiki/extensions/GrowthExperiments@master] Remove GEInfoboxTemplatesTest

https://gerrit.wikimedia.org/r/1113749

Maintenance_bot removed a project: Patch-For-Review.Jan 23 2025, 10:30 AM
gerritbot added a comment.Jan 23 2025, 10:43 AM

Change #1113760 had a related patch set uploaded (by Urbanecm; author: Urbanecm):

[mediawiki/extensions/GrowthExperiments@wmf/1.44.0-wmf.13] Remove GEInfoboxTemplatesTest

https://gerrit.wikimedia.org/r/1113760

gerritbot added a project: Patch-For-Review.Jan 23 2025, 10:43 AM

Change #1113761 had a related patch set uploaded (by Urbanecm; author: Urbanecm):

[mediawiki/extensions/GrowthExperiments@wmf/1.44.0-wmf.12] Remove GEInfoboxTemplatesTest

https://gerrit.wikimedia.org/r/1113761

ReleaseTaggerBot added a project: MW-1.44-notes (1.44.0-wmf.14; 2025-01-28).Jan 23 2025, 11:00 AM
gerritbot added a comment.Jan 23 2025, 11:09 AM

Change #1113760 merged by jenkins-bot:

[mediawiki/extensions/GrowthExperiments@wmf/1.44.0-wmf.13] Remove GEInfoboxTemplatesTest

https://gerrit.wikimedia.org/r/1113760

gerritbot added a comment.Jan 23 2025, 11:13 AM

Change #1113761 merged by Urbanecm:

[mediawiki/extensions/GrowthExperiments@wmf/1.44.0-wmf.12] Remove GEInfoboxTemplatesTest

https://gerrit.wikimedia.org/r/1113761

Stashbot added a comment.Jan 23 2025, 11:14 AM

Mentioned in SAL (#wikimedia-operations) [2025-01-23T11:14:12Z] <urbanecm@deploy2002> Started scap sync-world: Backport for [[gerrit:1113761|Remove GEInfoboxTemplatesTest (T384254)]], [[gerrit:1113760|Remove GEInfoboxTemplatesTest (T384254)]]

Stashbot added a comment.Jan 23 2025, 11:19 AM

Mentioned in SAL (#wikimedia-operations) [2025-01-23T11:19:43Z] <urbanecm@deploy2002> urbanecm: Backport for [[gerrit:1113761|Remove GEInfoboxTemplatesTest (T384254)]], [[gerrit:1113760|Remove GEInfoboxTemplatesTest (T384254)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)

Maintenance_bot removed a project: Patch-For-Review.Jan 23 2025, 11:30 AM
Stashbot added a comment.Jan 23 2025, 11:35 AM

Mentioned in SAL (#wikimedia-operations) [2025-01-23T11:35:45Z] <urbanecm@deploy2002> Started scap sync-world: Backport for [[gerrit:1113761|Remove GEInfoboxTemplatesTest (T384254)]], [[gerrit:1113760|Remove GEInfoboxTemplatesTest (T384254)]]

Urbanecm_WMF added a comment.Jan 23 2025, 11:51 AM
In T384254#10486160, @Michael wrote:

@Urbanecm_WMF Is GEInfoboxTemplatesTest actually still in use anywhere? Maybe the easiest is to just remove all of that.

To answer this question: It is (as in, you can access a feature that makes use of it), but it is not used in anything userfacing (searching for hastemplatecollection:infoboxtest would do that, same as hastemplatecollection:infobox, which actually is used to filter out articles with infoboxes). It was intended for testing the infobox detection feature. However, that period is long behind us, so there is no reason to keep the testing field.

Stashbot added a comment.Jan 23 2025, 11:51 AM

Mentioned in SAL (#wikimedia-operations) [2025-01-23T11:51:09Z] <urbanecm@deploy2002> Finished scap sync-world: Backport for [[gerrit:1113761|Remove GEInfoboxTemplatesTest (T384254)]], [[gerrit:1113760|Remove GEInfoboxTemplatesTest (T384254)]] (duration: 15m 23s)

Urbanecm_WMF closed this task as Resolved.Jan 23 2025, 11:55 AM

Should be resolved.

ReleaseTaggerBot edited projects, added: MW-1.44-notes (1.44.0-wmf.12; 2025-01-14); removed: MW-1.44-notes (1.44.0-wmf.14; 2025-01-28).Jan 23 2025, 12:00 PM
brennen added a comment.Jan 23 2025, 3:21 PM

Should be resolved.

Thanks!

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits