Background

CommunityConfiguration exposes the MediaWikiConfigReader class, which can be used to access community configuration as if it was a Config object (which can be injected to anything that accepts one). For example, it is possible to get the value of GEMentorshipAutomaticEligibility by calling MediaWikiConfigReader::get( 'GEMentorshipAutomaticEligibility' ). The idea is to allow MediaWikiConfigReader to act as a drop-in replacement for GlobalVarConfig (which is the default implementation).

Unfortunately config variables can be registered across various configuration providers, and it is not possible to determine where CommunityConfiguration should expect a configuration variable based on its name. However, config names are globally unique across all of MediaWiki stack (they are usually exposed as PHP globals after all), CommunityConfiguration maintains a map of configuration names to configuration providers.

For example, a portion of this map could look like: "GEMentorshipAutomaticEligibility" => "Mentorship", which would inform CommunityConfiguration that GEMentorshipAutomaticEligibility is expected to be found in the Mentorship configuration provider (ConfigurationProviderFactory::newProvider( 'Mentorship' )->get( 'GEMentorshipAutomaticEligibility' ) should return the desired value). This way, CommunityConfiguration does not have to iterate across all available providers and query them all before it eventually finds one that matches (or not); instead, the lookup can be done in constant time regardless of the number of providers.

To ensure it is reused whenever possible, CommunityConfiguration caches this map in local server object cache (which is backed by APCu) with an expiry time of TTL_DAY. CommunityConfiguration never attempts to invalidate this cache; this is done, because we assume the only way how the map could change is through a code deployment (which causes a rolling restart of the cluster, which causes the local server object cache to be purged [as it does not survive server restarts]).

Problem

SUL3 breaks the assumption CommunityConfiguration has, and it allows the configuration => provider map (described above) to change without a code deployment. This is because WMF wikis have significantly tightened configuration when they are accessed via the auth.wikimedia.org domain (many extensions are disabled, many features are disabled, etc.). In other words, a wiki can have a different map when accessed via the auth domain and when accessed normally. This results in cache pollution within the CommunityConfiguration extension.

Specifically, the issue can be reproduced by following the steps below:

1. Pick a non-k8s debug server (such as mwdebug2001).
2. Restart PHP-FPM on that server (sudo -i /usr/local/sbin/restart-php7.4-fpm), to ensure the APCu cache is empty. This is not necessary strictly speaking, but it practically rules out other causes of the error (as you are likely to be the only one hitting that cache).
3. Enable XWikimediaDebug for mwdebug2001.
4. Ensure you are logged out at test.wikipedia.org.
5. Login to any account with Growth features enabled (any new account will do): this will redirect you to auth.wikimedia.org (which is the tightened environment for SUL3 authentication) for the login form and then back to test.wikipedia.org (SUL3 kicks in)
   1. Incident-Reporting-System will query CommunityConfiguration (as part of the BeforePageDisplay hook).
   2. CommunityConfiguration will compute its map (as the cache is empty). It will not include any providers declared by GrowthExperiments, because GrowthExperiments is not enabled at auth.wikimedia.org (while both Incident-Reporting-System and MediaWiki-extensions-CommunityConfiguration are).
6. At this point, CommunityConfiguration's cache is corrupted (GrowthExperiments is enabled, as you left the authentication domain, but it is not included in CommunityConfiguration's map).
7. Open an editor of any page in the main space
   1. GrowthExperiments will query CommunityConfiguration
   2. CommunityConfiguration will reuse the map from step 5.B. It won't find the requested configuration variable there, and it will fail back to GlobalVarConfig.

For GrowthExperiments, this will result in the community configuration defined values not being respected. This is because we (still) have them in extension.json as well (as part of legacy CC code). Once we remove it (T367574), this would probably result in a fatal error instead (as GlobalVarConfig would not have the requested variable either).