Page MenuHomePhabricator

Flaky test WANObjectCacheTest::testPreemptiveRefresh (at least under php8.2 and php8.3)
Closed, ResolvedPublic

Description

It seems the following test can fail under php8.2+, but also passed the most time, it seems the chance to get flaky is higher in modern php

https://integration.wikimedia.org/ci/job/quibble-vendor-mysql-php82-noselenium-docker/146/console

1) WANObjectCacheTest::testPreemptiveRefresh
Value cached
Failed asserting that 2 is identical to 1.

/workspace/src/tests/phpunit/unit/includes/libs/objectcache/WANObjectCacheTest.php:658
/workspace/src/tests/phpunit/MediaWikiUnitTestCase.php:127

Passed under https://integration.wikimedia.org/ci/job/quibble-vendor-mysql-php82-noselenium-docker/126/console

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald Transcript

There is a mt_rand in WANObjectCache::worthRefreshExpiring and with php8.2 there is a new/refactored random extension.
I have seen this flaky test only on php8.2+, so this could be an issue that something change and now the random makes problems on CI.

php8.2 is now voting on core, if there are to many fails the test can be skipped by until the issue is investigated.

WANObjectCacheTest.php
	public function testPreemptiveRefresh() {
		$this->markTestSkippedIfPhp( '>=', '8.2' );

Change 983258 had a related patch set uploaded (by Jforrester; author: Jforrester):

[mediawiki/core@master] WANObjectCacheTest::testPreemptiveRefresh: Skip flaky test for PHP 8.2+

https://gerrit.wikimedia.org/r/983258

Change 983258 merged by jenkins-bot:

[mediawiki/core@master] WANObjectCacheTest::testPreemptiveRefresh: Skip flaky test for PHP 8.2+

https://gerrit.wikimedia.org/r/983258

Change #1017397 had a related patch set uploaded (by Reedy; author: Jforrester):

[mediawiki/core@REL1_41] WANObjectCacheTest::testPreemptiveRefresh: Skip flaky test for PHP 8.2+

https://gerrit.wikimedia.org/r/1017397

Change #1017397 merged by jenkins-bot:

[mediawiki/core@REL1_41] WANObjectCacheTest::testPreemptiveRefresh: Skip flaky test for PHP 8.2+

https://gerrit.wikimedia.org/r/1017397

Krinkle triaged this task as Medium priority.Oct 28 2025, 8:44 PM
--- a/tests/phpunit/unit/includes/libs/objectcache/WANObjectCacheTest.php
+++ b/tests/phpunit/unit/includes/libs/objectcache/WANObjectCacheTest.php
@@ -581,7 +581,7 @@ class WANObjectCacheTest extends MediaWikiUnitTestCase {
 
    public function testPreemptiveRefresh() {
        // (T353180) Flaky test, to fix and re-enable
-       self::markTestSkippedIfPhp( '>=', '8.2' );
+       // self::markTestSkippedIfPhp( '>=', '8.2' );
 
        $value = 'KatCafe';
        $wasSet = 0;

When re-enabling this test, I can reproduce the failure locally on PHP 8.3. When repeating the below 10-20 times in a row, one or two will fail.

$ composer phpunit tests/phpunit/unit/includes/libs/objectcache/
Using PHP 8.3.24
Running without MediaWiki settings because there are no integration tests
PHPUnit 9.6.21 by Sebastian Bergmann and contributors.

.....................................................F.........  63 / 123 ( 51%)
............................................................    123 / 123 (100%)

Time: 00:00.047, Memory: 30.64 MB

There was 1 failure:

1) Wikimedia\Tests\ObjectCache\WANObjectCacheTest::testPreemptiveRefresh
Value cached
Failed asserting that 2 is identical to 1.

/Users/krinkle/Development/mediawiki/tests/phpunit/unit/includes/libs/objectcache/WANObjectCacheTest.php:671

Debugging:

--- a/includes/libs/objectcache/WANObjectCache.php
+++ b/includes/libs/objectcache/WANObjectCache.php
@@ -2709,6 +2709,7 @@ class WANObjectCache implements
 		// Ramp up $chance from 0 to its nominal value over RAMPUP_TTL seconds to avoid stampedes
 		$chance *= ( $timeOld <= self::RAMPUP_TTL ) ? $timeOld / self::RAMPUP_TTL : 1;
 
+		$this->logger->debug( "worthRefreshPopular: $chance chance" );
 		return ( mt_rand( 1, 1_000_000_000 ) <= 1_000_000_000 * $chance );
 	}
 
(1/2) Stage this hunk [y,n,q,a,d,j,J,g,/,e,?]? y
@@ -2769,6 +2770,7 @@ class WANObjectCache implements
 		// having p(0)=0 and p(1)=1. The value expires at the nominal expiry.
 		$chance = $ttrRatio ** 4;
 
+		$this->logger->debug( "worthRefreshExpiring: $chance chance" );
 		return ( mt_rand( 1, 1_000_000_000 ) <= 1_000_000_000 * $chance );
 	}
 
(2/2) Stage this hunk [y,n,q,a,d,K,g,/,e,?]? y

diff --git a/tests/phpunit/unit/includes/libs/objectcache/WANObjectCacheTest.php b/tests/phpunit/unit/includes/libs/objectcache/WANObjectCacheTest.php
index 0742d78894..1dc3b2aab9 100644
--- a/tests/phpunit/unit/includes/libs/objectcache/WANObjectCacheTest.php
+++ b/tests/phpunit/unit/includes/libs/objectcache/WANObjectCacheTest.php
@@ -651,7 +651,8 @@ class WANObjectCacheTest extends MediaWikiUnitTestCase {
 		$this->assertSame( $value, $v, "New value returned" );
 
 		$cache = new PopularityRefreshingWANObjectCache( [
-			'cache'   => new HashBagOStuff()
+			'cache'   => new HashBagOStuff(),
+			'logger' => new \MediaWiki\Logger\ConsoleLogger( 'objectcache' )
 		] );
 		$mockWallClock = 1549343530.0;
 		$cache->setMockTime( $mockWallClock );
When it passes
$ composer phpunit tests/phpunit/unit/includes/libs/objectcache/WANObjectCacheTest.php 
Using PHP 8.3.24
Running without MediaWiki settings because there are no integration tests
PHPUnit 9.6.21 by Sebastian Bergmann and contributors.

............................[debug] [objectcache] fetchOrRegenerate(tiffany-breakfast:too-new-for-refresh): miss, new value computed
[debug] [objectcache] worthRefreshExpiring: 0.0625 chance
..................................... 65 / 98 ( 66%)
.................................                                 98 / 98 (100%)

Time: 00:00.043, Memory: 30.64 MB

OK (98 tests, 991 assertions)
When it fails
$ composer phpunit tests/phpunit/unit/includes/libs/objectcache/WANObjectCacheTest.php 
Using PHP 8.3.24
Running without MediaWiki settings because there are no integration tests
PHPUnit 9.6.21 by Sebastian Bergmann and contributors.

............................[debug] [objectcache] fetchOrRegenerate(tiffany-breakfast:too-new-for-refresh): miss, new value computed
[debug] [objectcache] worthRefreshExpiring: 0.0625 chance
[debug] [objectcache] fetchOrRegenerate(tiffany-breakfast:too-new-for-refresh): hit with sync refresh
[debug] [objectcache] fetchOrRegenerate(tiffany-breakfast:too-new-for-refresh): miss, new value computed
F.................................... 65 / 98 ( 66%)
.................................                                 98 / 98 (100%)

Time: 00:00.043, Memory: 30.64 MB

There was 1 failure:

1) Wikimedia\Tests\ObjectCache\WANObjectCacheTest::testPreemptiveRefresh
Value cached
Failed asserting that 2 is identical to 1.

/Users/krinkle/Development/mediawiki/tests/phpunit/unit/includes/libs/objectcache/WANObjectCacheTest.php:668

Looks like the test for hotTTR correctly mocks the hotTTR random chance, but doesn't mock the lowTTL random chance, and that's causing it to fail 6.25% of the time. That translates to 1:16 times, and matches "one in 10-20 times" failure we've been seeing.

Change #1199543 had a related patch set uploaded (by Krinkle; author: Krinkle):

[mediawiki/core@master] objectcache: Improve explanation of lowTTL and hotTTR and async regens

https://gerrit.wikimedia.org/r/1199543

Change #1199544 had a related patch set uploaded (by Krinkle; author: Krinkle):

[mediawiki/core@master] objectcache: Fix flaky testPreemptiveRefresh

https://gerrit.wikimedia.org/r/1199544

Change #1199545 had a related patch set uploaded (by Krinkle; author: Krinkle):

[mediawiki/core@master] objectcache: Improve and simplify testPreemptiveRefresh

https://gerrit.wikimedia.org/r/1199545

Change #1199543 merged by jenkins-bot:

[mediawiki/core@master] objectcache: Improve explanation of lowTTL and hotTTR and async regens

https://gerrit.wikimedia.org/r/1199543

Change #1199544 merged by jenkins-bot:

[mediawiki/core@master] objectcache: Fix flaky testPreemptiveRefresh

https://gerrit.wikimedia.org/r/1199544

Change #1199545 merged by jenkins-bot:

[mediawiki/core@master] objectcache: Improve and simplify testPreemptiveRefresh

https://gerrit.wikimedia.org/r/1199545

DAlangi_WMF subscribed.

All patches merged and tests fixed. Thanks, @Krinkle, for working on this.