My understanding is that the dblist is used on InitialiseSettings and/or CommonSettings, if you don't touch those the change doesn't get applied as some sort of 'caching' issue. Fact is that after touching InitialiseSettings the dblist patch merged previously started to work as expected. Thanks for filing this task.

The dblist files are read and are part of the cached config. The relevant code (inline annotations are mine):

// File in which we cache the wiki configuration
 188 $filename = "/tmp/mw-cache-$wmgVersionNumber/conf-$wgDBname";
 189 if ( defined( 'HHVM_VERSION' ) ) {
 190     $filename .= '-hhvm';
 191 }
 192 

// $globals would hold the values
 193 $globals = false;

// load from cache unless InitialiseSettings.php is more recent. Or to say otherwise, to dismiss the cache, InitialiseSettings.php has to be touched.
 194 if ( @filemtime( $filename ) >= filemtime( "$wmfConfigDir/InitialiseSettings.php" ) ) {
 195     $cacheRecord = @file_get_contents( $filename );
 196     if ( $cacheRecord !== false ) {
 197         $globals = unserialize( $cacheRecord );
 198     }
 199 }
 200 
 201 if ( !$globals ) {

// cache not loaded so process 
202     # Get configuration from SiteConfiguration object
 203     require "$wmfConfigDir/InitialiseSettings.php";
 204 

// And here we read the .dblist files
 205     # Collect all the dblist tags associated with this wiki
 206     $wikiTags = [];
 207     # When updating list please run ./docroot/noc/createTxtFileSymlinks.sh
 208     # Expand computed dblists with ./multiversion/bin/expanddblist
 209     foreach ( [
 210         'private',
...
 243         'pp_stage0',
 244         'pp_stage1',
 245         'cirrussearch-big-indices',
 246     ] as $tag ) {
 247         $dblist = MWWikiversions::readDbListFile( $tag );
 248         if ( in_array( $wgDBname, $dblist ) ) {
 249             $wikiTags[] = $tag;
 250         }
 251     }

Conclusion: the .dblist files are read and cached in the global wiki configuration. To invalidate them one has to touch InitialiseSettings.php and deploy it. Maybe scap can be made smart enough and automatically touch InitialiseSettings.php when it is asked to sync a file in ./dblists/.

I think I have a theory, and I think it could explain how this keeps happening.

Here's a diagram:

How do we fix this? We could set the mtime way in the future in scap, I guess. This system is really hard to reason about as it is and I'd be happier with a different solution.

In T217830#5009234, @thcipriani wrote:

How do we fix this? We could set the mtime way in the future in scap, I guess. This system is really hard to reason about as it is and I'd be happier with a different solution.

@hashar suggested in IRC we have scap generate the cache rather than touch InitialiseSettings.php and have the application generate the cache (the current situation). We'd have to do that for all databases for all active versions, which might be time consuming, but it would side-step this problem.

Is this cache still needed with php7? could we get rid of the cache? I don't know who can answer that...

From git log the filemtime has been there since at least 2011 / MediaWiki 1.17. We had php 5.2 at that time Special:Version in June 2011.

So yeah it is probably way faster to just reprocess it now thanks to hhvm/php7.0? But we would need a feature flag and some way to measure the difference with and without filemtime and the serialized configs. Maybe Performance-Team team could be interested in tackling/helping on that front?

I am tempted to just merge this as a duplicate of the old task T181833 (and maybe update its description with Tyler comments above).

For the usefulness of filemtime and the serialized cache of the configuration, that predates our time by years. It has been there since at least June 2011 when we ran php 5.2. Supposedly nowadays with HHVM / PHP 7.0 parsing the php file might not be much of a problem anywere. And might well be faster since the PHP is read once and held in a bytecode cache, saving PHP from having to read file and unserialize. But I think that is a subject for another task.

I think the simpler solution would be to do the same as what we generally to avoid this problem in production code, e.g. in MediaWiki core. Which is to have the verification mechanism part of the value instead of as some fragile indirect side-effect of the value.

In other words, either

1. Make the fragile side-effect not a side effect and not fragile, by having the PHP code in wmf-config create the cache with the explicit mtime set to the value it was creating it for.
2. Or, store the target mtime in the cache value and verify it on retrieval. Thus removing use of filemtime for the cache file.

In T217830#5178490, @Krinkle wrote:

I think the simpler solution would be to do the same as what we generally to avoid this problem in production code, e.g. in MediaWiki core. Which is to have the verification mechanism part of the value instead of as some fragile indirect side-effect of the value.

In other words, either

1. Make the fragile side-effect not a side effect and not fragile, by having the PHP code in wmf-config create the cache with the explicit mtime set to the value it was creating it for.

This is what I expected the code did; i.e., generate a /tmp cache file, set mtime on cache file to the mtime of IS.php, mv cachefile into place -- this would side-step the race-condition, I believe.

2. Or, store the target mtime in the cache value and verify it on retrieval. Thus removing use of filemtime for the cache file.

Change 528446 had a related patch set uploaded (by Krinkle; owner: Krinkle):
[operations/mediawiki-config@master] CommonSettings: Clean up wmf-config caching code [no-op]

https://gerrit.wikimedia.org/r/528446

Change 528447 had a related patch set uploaded (by Krinkle; owner: Krinkle):
[operations/mediawiki-config@master] CommonSettings: Store mtime inside wmf-config cache file

https://gerrit.wikimedia.org/r/528447

Change 528446 merged by jenkins-bot:
[operations/mediawiki-config@master] CommonSettings: Clean up wmf-config caching code [no-op]

https://gerrit.wikimedia.org/r/528446

Change 528447 merged by jenkins-bot:
[operations/mediawiki-config@master] CommonSettings: Store mtime inside wmf-config cache file

https://gerrit.wikimedia.org/r/528447

Mentioned in SAL (#wikimedia-operations) [2019-09-04T01:02:06Z] <krinkle@deploy1001> Synchronized wmf-config/CommonSettings.php: ed5297c10 / T217830 (duration: 00m 59s)

Is there anything more to do here?