Investigate seemingly-impossible "ConfigException: undefined option" in minutes after deployment
Open, Needs TriagePublicPRODUCTION ERROR
Actions

Assigned To

None

Authored By

	TheresNoTime
	Nov 3 2022, 8:31 PM

Project Tags

Referenced Files

	F35791490: Screenshot 2022-11-17 at 16.43.42.png
	Nov 17 2022, 4:45 PM

Subscribers

Description

Error

mwversion: 1.40.0-wmf.8
reqId: e4ad1ce0-6c5e-4e64-94cb-9e30706f0dad
Find reqId in Logstash

normalized_message

[{reqId}] {exception_url}   ConfigException: GlobalVarConfig::get: undefined option: 'VectorArticleTools'

exception.trace

from /srv/mediawiki/php-1.40.0-wmf.8/includes/config/GlobalVarConfig.php(59)
#0 /srv/mediawiki/php-1.40.0-wmf.8/skins/Vector/includes/FeatureManagement/Requirements/OverridableConfigRequirement.php(184): GlobalVarConfig->get(string)
#1 /srv/mediawiki/php-1.40.0-wmf.8/skins/Vector/includes/FeatureManagement/FeatureManager.php(160): MediaWiki\Skins\Vector\FeatureManagement\Requirements\OverridableConfigRequirement->isMet()
#2 /srv/mediawiki/php-1.40.0-wmf.8/skins/Vector/includes/FeatureManagement/FeatureManager.php(140): MediaWiki\Skins\Vector\FeatureManagement\FeatureManager->isFeatureEnabled(string)
#3 [internal function]: MediaWiki\Skins\Vector\FeatureManagement\FeatureManager::MediaWiki\Skins\Vector\FeatureManagement\{closure}(string)
#4 /srv/mediawiki/php-1.40.0-wmf.8/skins/Vector/includes/FeatureManagement/FeatureManager.php(141): array_map(Closure, array)
#5 /srv/mediawiki/php-1.40.0-wmf.8/skins/Vector/includes/Hooks.php(725): MediaWiki\Skins\Vector\FeatureManagement\FeatureManager->getFeatureBodyClass()
#6 /srv/mediawiki/php-1.40.0-wmf.8/includes/HookContainer/HookContainer.php(160): MediaWiki\Skins\Vector\Hooks->onOutputPageBodyAttributes(OutputPage, MediaWiki\Skins\Vector\SkinVector22, array)
#7 /srv/mediawiki/php-1.40.0-wmf.8/includes/HookContainer/HookRunner.php(2712): MediaWiki\HookContainer\HookContainer->run(string, array, array)
#8 /srv/mediawiki/php-1.40.0-wmf.8/includes/OutputPage.php(3437): MediaWiki\HookContainer\HookRunner->onOutputPageBodyAttributes(OutputPage, MediaWiki\Skins\Vector\SkinVector22, array)
#9 /srv/mediawiki/php-1.40.0-wmf.8/includes/skins/SkinMustache.php(63): OutputPage->headElement(MediaWiki\Skins\Vector\SkinVector22)
#10 /srv/mediawiki/php-1.40.0-wmf.8/includes/skins/SkinTemplate.php(177): SkinMustache->generateHTML()
#11 /srv/mediawiki/php-1.40.0-wmf.8/includes/OutputPage.php(2893): SkinTemplate->outputPage()
#12 /srv/mediawiki/php-1.40.0-wmf.8/includes/MediaWiki.php(923): OutputPage->output(boolean)
#13 /srv/mediawiki/php-1.40.0-wmf.8/includes/MediaWiki.php(563): MediaWiki->main()
#14 /srv/mediawiki/php-1.40.0-wmf.8/index.php(50): MediaWiki->run()
#15 /srv/mediawiki/php-1.40.0-wmf.8/index.php(46): wfIndexMain()
#16 /srv/mediawiki/w/index.php(3): require(string)
#17 {main}

Impact

Notes

High spike during deployment of https://gerrit.wikimedia.org/r/c/852882/

Details

Request URL: https://ko.wikipedia.org/wiki/I_Was_the_One

Event Timeline

TheresNoTime created this task.Nov 3 2022, 8:31 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptNov 3 2022, 8:31 PM

TheresNoTime updated the task description. (Show Details)Nov 3 2022, 8:33 PM

TheresNoTime added a project: Vector 2022.

Interesting. Looks similar to a sync ordering problem. That is, the code was somehow still looking for VectorArticleTools even after these patches were deployed. We restart php-fpm now as a part of deployment, so that should clear the opcache and trigger a re-read of the php files on disk.

However, I suppose, the question is how the variables from skin.json become global(ish) variables and is that still an edge-case?

I note in this instance, the spike seemed to clear up on its own (which was not the case when we saw things like this previously, so that's good)

Jdlrobson moved this task from Incoming to Work with other teams on the Vector 2022 board.Nov 7 2022, 3:27 PM

Krinkle moved this task from Untriaged to Nov 2022 on the Wikimedia-production-error board.Nov 17 2022, 4:37 PM

It affected all wikis, and most or all mw* servers in all clusters (web appserver, api appserver, parsoid, other/wikitech), including the canaries in a blib shortly before the others:

Screenshot 2022-11-17 at 16.43.42.png (1,910×982 px, 187 KB)

Krinkle renamed this task from ConfigException: GlobalVarConfig::get: undefined option: 'VectorArticleTools' to Investigate seemingly-impossible "ConfigException: undefined option" in minutes after deployment.Nov 17 2022, 4:45 PM

Krinkle edited projects, added: Sustainability (Incident Followup), MediaWiki-Configuration, Deployments; removed: Vector 2022, Wikimedia-production-error.

Investigate seemingly-impossible "ConfigException: undefined option" in minutes after deploymentOpen, Needs TriagePublicPRODUCTION ERRORActions