Page MenuHomePhabricator

Beta cluster log spam: MediaWiki\Extension\CommunityConfiguration\Access\MediaWikiConfigReader was unable to find GEInfoboxTemplatesTest in community configuration, returning configuration from the fallback config
Closed, ResolvedPublic

Description

Very high rate starting from today, noticed while browsing the logs for another error.

MediaWiki\Extension\CommunityConfiguration\Access\MediaWikiConfigReader was unable to find GEInfoboxTemplatesTest in community configuration, returning configuration from the fallback config

from /srv/mediawiki/php-master/extensions/CommunityConfiguration/src/Access/MediaWikiConfigReader.php(120)
#0 /srv/mediawiki/php-master/extensions/CommunityConfiguration/src/Access/MediaWikiConfigReader.php(131): MediaWiki\Extension\CommunityConfiguration\Access\MediaWikiConfigReader->getConfigByVariableName(string)
#1 /srv/mediawiki/php-master/extensions/GrowthExperiments/includes/Config/MediaWikiConfigReaderWrapper.php(46): MediaWiki\Extension\CommunityConfiguration\Access\MediaWikiConfigReader->get(string)
#2 /srv/mediawiki/php-master/extensions/GrowthExperiments/includes/HomepageHooks.php(1409): GrowthExperiments\Config\MediaWikiConfigReaderWrapper->get(string)
#3 /srv/mediawiki/php-master/includes/HookContainer/HookContainer.php(155): GrowthExperiments\HomepageHooks::onCirrusSearchAddQueryFeatures(CirrusSearch\SearchConfig, array)
#4 /srv/mediawiki/php-master/extensions/CirrusSearch/includes/CirrusSearchHookRunner.php(33): MediaWiki\HookContainer\HookContainer->run(string, array)
#5 /srv/mediawiki/php-master/extensions/CirrusSearch/includes/Parser/FullTextKeywordRegistry.php(99): CirrusSearch\CirrusSearchHookRunner->onCirrusSearchAddQueryFeatures(CirrusSearch\SearchConfig, array)
#6 /srv/mediawiki/php-master/extensions/CirrusSearch/includes/Searcher.php(308): CirrusSearch\Parser\FullTextKeywordRegistry->__construct(CirrusSearch\SearchConfig)
#7 /srv/mediawiki/php-master/extensions/CirrusSearch/includes/Searcher.php(340): CirrusSearch\Searcher->buildFullTextSearch(string)
#8 /srv/mediawiki/php-master/extensions/CirrusSearch/includes/Searcher.php(219): CirrusSearch\Searcher->searchTextInternal(string)
#9 /srv/mediawiki/php-master/extensions/CirrusSearch/includes/CirrusSearch.php(305): CirrusSearch\Searcher->search(CirrusSearch\Search\SearchQuery)
#10 /srv/mediawiki/php-master/extensions/CirrusSearch/includes/CirrusSearch.php(278): CirrusSearch\CirrusSearch->searchTextReal(CirrusSearch\Search\SearchQuery)
#11 /srv/mediawiki/php-master/includes/search/SearchEngine.php(102): CirrusSearch\CirrusSearch->doSearchText(string)
#12 /srv/mediawiki/php-master/includes/search/SearchEngine.php(194): SearchEngine->{closure}()
#13 /srv/mediawiki/php-master/includes/search/SearchEngine.php(103): SearchEngine->maybePaginate(Closure)
#14 /srv/mediawiki/php-master/extensions/GrowthExperiments/includes/NewcomerTasks/TaskSuggester/LocalSearchTaskSuggester.php(130): SearchEngine->searchText(string)
#15 /srv/mediawiki/php-master/extensions/GrowthExperiments/includes/NewcomerTasks/TaskSuggester/SearchTaskSuggester.php(196): GrowthExperiments\NewcomerTasks\TaskSuggester\LocalSearchTaskSuggester->search(GrowthExperiments\NewcomerTasks\TaskSuggester\SearchStrategy\SearchQuery, int, int, bool)
#16 /srv/mediawiki/php-master/extensions/GrowthExperiments/includes/NewcomerTasks/TaskSuggester/SearchTaskSuggester.php(92): GrowthExperiments\NewcomerTasks\TaskSuggester\SearchTaskSuggester->doSuggest(null, MediaWiki\User\User, GrowthExperiments\NewcomerTasks\Task\TaskSetFilters, int, int, array)
#17 /srv/mediawiki/php-master/extensions/GrowthExperiments/includes/NewcomerTasks/TaskSuggester/LocalSearchTaskSuggester.php(73): GrowthExperiments\NewcomerTasks\TaskSuggester\SearchTaskSuggester->suggest(MediaWiki\User\User, GrowthExperiments\NewcomerTasks\Task\TaskSetFilters, int, null, array)
#18 /srv/mediawiki/php-master/extensions/GrowthExperiments/includes/NewcomerTasks/TaskSuggester/CacheDecorator.php(80): GrowthExperiments\NewcomerTasks\TaskSuggester\LocalSearchTaskSuggester->suggest(MediaWiki\User\User, GrowthExperiments\NewcomerTasks\Task\TaskSetFilters, int, null, array)
#19 /srv/mediawiki/php-master/extensions/GrowthExperiments/includes/NewcomerTasks/TaskSuggester/QualityGateDecorator.php(82): GrowthExperiments\NewcomerTasks\TaskSuggester\CacheDecorator->suggest(MediaWiki\User\User, GrowthExperiments\NewcomerTasks\Task\TaskSetFilters, int, null, array)
#20 /srv/mediawiki/php-master/extensions/GrowthExperiments/maintenance/refreshLinkRecommendations.php(245): GrowthExperiments\NewcomerTasks\TaskSuggester\QualityGateDecorator->suggest(MediaWiki\User\User, GrowthExperiments\NewcomerTasks\Task\TaskSetFilters, int, null, array)
#21 /srv/mediawiki/php-master/extensions/GrowthExperiments/maintenance/refreshLinkRecommendations.php(140): GrowthExperiments\Maintenance\RefreshLinkRecommendations->findArticlesInTopic(string)
#22 /srv/mediawiki/php-master/maintenance/includes/MaintenanceRunner.php(695): GrowthExperiments\Maintenance\RefreshLinkRecommendations->execute()
#23 /srv/mediawiki/php-master/maintenance/run.php(51): MediaWiki\Maintenance\MaintenanceRunner->run()
#24 /srv/mediawiki/multiversion/MWScript.php(156): require_once(string)
#25 {main}

https://beta-logs.wmcloud.org/goto/e9a1972b4cf937a783efb3a09cfcc4a0

image.png (532×2 px, 73 KB)

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald Transcript

FTR, this is likely to become production spam when the train moves forward.

brennen subscribed.

FTR, this is likely to become production spam when the train moves forward.

FWIW, I did not see this when moving the train to group0 yesterday. Do you think it's likely to be an issue at group1 and beyond?

FTR, this is likely to become production spam when the train moves forward.

FWIW, I did not see this when moving the train to group0 yesterday.

Really? I checked in production, and I do see those events coming from group0 at a fairly high volume (CommunityConfiguration.log has 250M+ of data for today, over 48k of log entries). Note group0 equals testwiki in this case, and since the events appear to be indeed appearing, their group2 volume has a chance of knocking the logging infrastructure down. FWIW, those are not errors, but logged at the INFO level (but at a fairly high rate).

Do you think it's likely to be an issue at group1 and beyond?

I wouldn't expect this to be lot worse for group1, Growth targets Wikipedias only, and there isn't a lot of them on group1. However, unless my imagination is very vivid and I'm seeing logs that are not being generated, I wouldn't recommend proceeding with group2.

brennen raised the priority of this task from High to Unbreak Now!.Wed, Jan 22, 7:27 PM

Really? I checked in production, and I do see those events coming from group0 at a fairly high volume (CommunityConfiguration.log has 250M+ of data for today, over 48k of log entries). Note group0 equals testwiki in this case, and since the events appear to be indeed appearing, their group2 volume has a chance of knocking the logging infrastructure down. FWIW, those are not errors, but logged at the INFO level (but at a fairly high rate).

Ah, I see. These don't hit the places we typically monitor for the train: mediawiki-errors and "MediaWiki New Errors ECS" dashboards and the log files consolidated by logspam-watch (exception.log, error.log).

I wouldn't expect this to be lot worse for group1, Growth targets Wikipedias only, and there isn't a lot of them on group1. However, unless my imagination is very vivid and I'm seeing logs that are not being generated, I wouldn't recommend proceeding with group2.

Ok, I'm raising priority here accordingly. Seeking some further input, but I will either:

  1. Roll to group1 but block for group2, on the theory that this will be basically status quo for this group and we're better off finding other things broken
  2. Roll back, on the theory that this is more log volume than we should treat as acceptable, since it eats into our margin for other problems

@Urbanecm_WMF Is GEInfoboxTemplatesTest actually still in use anywhere? Maybe the easiest is to just remove all of that.

I tried code-search for GEInfoboxTemplatesTest and code-search for infoboxtest. And only GrowthExperiments shows up.

Alternatively, we could probably revert MediaWikiConfigReader: Log when falling back which should also be straight forward and basically without downsides as this only added the logging that is now causing problems.

EDIT: Then again, the description for that config is "List of templates that provide Infobox functionality. Used for manually testing changes.", so maybe we don't want to throw it out on a whim. That would mean that reverting the above change might be the right way to mitigate this train-blocker, and then we can think about what we want to do about that config.

Talked over with @jeena as backup. For simplicity's sake we're just going to block here.

Change #1113749 had a related patch set uploaded (by Urbanecm; author: Urbanecm):

[mediawiki/extensions/GrowthExperiments@master] Remove GEInfoboxTemplatesTest

https://gerrit.wikimedia.org/r/1113749

Change #1113749 merged by jenkins-bot:

[mediawiki/extensions/GrowthExperiments@master] Remove GEInfoboxTemplatesTest

https://gerrit.wikimedia.org/r/1113749

Change #1113760 had a related patch set uploaded (by Urbanecm; author: Urbanecm):

[mediawiki/extensions/GrowthExperiments@wmf/1.44.0-wmf.13] Remove GEInfoboxTemplatesTest

https://gerrit.wikimedia.org/r/1113760

Change #1113761 had a related patch set uploaded (by Urbanecm; author: Urbanecm):

[mediawiki/extensions/GrowthExperiments@wmf/1.44.0-wmf.12] Remove GEInfoboxTemplatesTest

https://gerrit.wikimedia.org/r/1113761

Change #1113760 merged by jenkins-bot:

[mediawiki/extensions/GrowthExperiments@wmf/1.44.0-wmf.13] Remove GEInfoboxTemplatesTest

https://gerrit.wikimedia.org/r/1113760

Change #1113761 merged by Urbanecm:

[mediawiki/extensions/GrowthExperiments@wmf/1.44.0-wmf.12] Remove GEInfoboxTemplatesTest

https://gerrit.wikimedia.org/r/1113761

Mentioned in SAL (#wikimedia-operations) [2025-01-23T11:14:12Z] <urbanecm@deploy2002> Started scap sync-world: Backport for [[gerrit:1113761|Remove GEInfoboxTemplatesTest (T384254)]], [[gerrit:1113760|Remove GEInfoboxTemplatesTest (T384254)]]

Mentioned in SAL (#wikimedia-operations) [2025-01-23T11:19:43Z] <urbanecm@deploy2002> urbanecm: Backport for [[gerrit:1113761|Remove GEInfoboxTemplatesTest (T384254)]], [[gerrit:1113760|Remove GEInfoboxTemplatesTest (T384254)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)

Mentioned in SAL (#wikimedia-operations) [2025-01-23T11:35:45Z] <urbanecm@deploy2002> Started scap sync-world: Backport for [[gerrit:1113761|Remove GEInfoboxTemplatesTest (T384254)]], [[gerrit:1113760|Remove GEInfoboxTemplatesTest (T384254)]]

@Urbanecm_WMF Is GEInfoboxTemplatesTest actually still in use anywhere? Maybe the easiest is to just remove all of that.

To answer this question: It is (as in, you can access a feature that makes use of it), but it is not used in anything userfacing (searching for hastemplatecollection:infoboxtest would do that, same as hastemplatecollection:infobox, which actually is used to filter out articles with infoboxes). It was intended for testing the infobox detection feature. However, that period is long behind us, so there is no reason to keep the testing field.

Mentioned in SAL (#wikimedia-operations) [2025-01-23T11:51:09Z] <urbanecm@deploy2002> Finished scap sync-world: Backport for [[gerrit:1113761|Remove GEInfoboxTemplatesTest (T384254)]], [[gerrit:1113760|Remove GEInfoboxTemplatesTest (T384254)]] (duration: 15m 23s)