Page MenuHomePhabricator
Create Task
Maniphest T422112

PHP Warning: Trying to access array offset on null
Closed, ResolvedPublicPRODUCTION ERROR
Actions

Description

Error
  • service.version: 1.46.0-wmf.22
  • timestamp: 2026-04-02T07:28:41.663Z
  • labels.phpversion: 8.3.30
  • trace.id: 763eaba4-2d9f-455e-9680-e345e6ce71ff
  • Find trace.id in Logstash
labels.normalized_message
[{reqId}] {exception_url}   PHP Warning: Trying to access array offset on null
FrameLocationCall
from/srv/mediawiki/php-1.46.0-wmf.22/extensions/TestKitchen/includes/Sdk/ExperimentManager.php(197)
#0/srv/mediawiki/php-1.46.0-wmf.22/extensions/TestKitchen/includes/Sdk/ExperimentManager.php(197)MediaWiki\Exception\MWExceptionHandler::handleError(int, string, string, int)
#1/srv/mediawiki/php-1.46.0-wmf.22/extensions/TestKitchen/includes/Sdk/ExperimentManager.php(152)MediaWiki\Extension\TestKitchen\Sdk\ExperimentManager->getExperimentConfig(string, array)
#2/srv/mediawiki/php-1.46.0-wmf.22/skins/MinervaNeue/includes/Skins/SkinMinerva.php(450)MediaWiki\Extension\TestKitchen\Sdk\ExperimentManager->getExperiment(string)
#3/srv/mediawiki/php-1.46.0-wmf.22/skins/MinervaNeue/includes/Skins/SkinMinerva.php(471)MediaWiki\Minerva\Skins\SkinMinerva->shouldShowAccountMenuItems()
#4/srv/mediawiki/php-1.46.0-wmf.22/skins/MinervaNeue/includes/Skins/SkinMinerva.php(309)MediaWiki\Minerva\Skins\SkinMinerva->getPersonalToolsMenu(array)
#5/srv/mediawiki/php-1.46.0-wmf.22/includes/Skin/SkinMustache.php(51)MediaWiki\Minerva\Skins\SkinMinerva->getTemplateData()
#6/srv/mediawiki/php-1.46.0-wmf.22/includes/Skin/SkinTemplate.php(172)MediaWiki\Skin\SkinMustache->generateHTML()
#7/srv/mediawiki/php-1.46.0-wmf.22/includes/Skin/Skin.php(671)MediaWiki\Skin\SkinTemplate->outputPage()
#8/srv/mediawiki/php-1.46.0-wmf.22/includes/Output/OutputPage.php(3313)MediaWiki\Skin\Skin->outputPageFinal(MediaWiki\Output\OutputPage)
#9/srv/mediawiki/php-1.46.0-wmf.22/includes/Actions/ActionEntryPoint.php(162)MediaWiki\Output\OutputPage->output(bool)
#10/srv/mediawiki/php-1.46.0-wmf.22/includes/MediaWikiEntryPoint.php(180)MediaWiki\Actions\ActionEntryPoint->execute()
#11/srv/mediawiki/php-1.46.0-wmf.22/index.php(44)MediaWiki\MediaWikiEntryPoint->run()
#12/srv/mediawiki/w/index.php(3)require(string)
#13{main}
Notes
  • Happens in bursts
  • Started with 1.46.0-wmf.22
  • Affects only the hebrew wiki so far After deployment to group2, now other wikis are also affected
  • Warnings are emitted at all 3 non-null-coalesced $experiments[ $experimentName ][ ... ] expressions in ExperimentManager::getExperimentConfig (L197, L205, L207), so you'll see a mix of source lines at the top of the stack (together with the associated PHP Warning: Undefined array key warnings for the specific $experimentNames not present in $experiments).

The bursts are big but short-lived. I'm monitoring this but not marking as blocker yet

Details

Request URL
https://he.wikipedia.org/wiki/%D7%92%27%D7%99%D7%99%D7%9F_%D7%9E%D7%A0%D7%A1%D7%A4%D7%99%D7%9C%D7%93
Related Changes in Gerrit:
SubjectAuthorRepoBranchLines +/-
PHP SDK: Split measurement of unknown experimentsPhuedxmediawiki/extensions/TestKitchenwmf/1.46.0-wmf.24+8 -5
PHP SDK: Split measurement of unknown experimentsPhuedxmediawiki/extensions/TestKitchenmaster+8 -5
ConfigsFetcher: Track cache/stash hit ratePhuedxmediawiki/extensions/TestKitchenmaster+38 -3
PHP SDK: Measure known experiments correctlyPhuedxmediawiki/extensions/TestKitchenwmf/1.46.0-wmf.22+63 -5
PHP SDK: Measure known experiments correctlyPhuedxmediawiki/extensions/TestKitchenwmf/1.46.0-wmf.23+63 -5
PHP SDK: Measure known experiments correctlyPhuedxmediawiki/extensions/TestKitchenmaster+63 -5
PHP SDK: Handle experiment config missing or malformedSantiago Facimediawiki/extensions/TestKitchenwmf/1.46.0-wmf.23+152 -80
PHP SDK: Handle experiment config missing or malformedSantiago Facimediawiki/extensions/TestKitchenwmf/1.46.0-wmf.22+152 -80
PHP SDK: Handle experiment config missing or malformedPhuedxmediawiki/extensions/TestKitchenmaster+152 -80
Customize query in gerrit

Related Objects
Search...

Event Timeline

jnuche created this task.Apr 2 2026, 8:18 AM
Restricted Application added a project: Growth-Team. · View Herald TranscriptApr 2 2026, 8:18 AM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
jnuche updated the task description. (Show Details)Apr 2 2026, 8:26 AM
jnuche added a comment.Apr 2 2026, 10:01 AM

@Sfaci @cjming In the last two hours there where 2 big bursts of >15K each.

Right now I don't think this warrants a rollback of the train, but the amount of noise we are introducing to the logs means we probably want to look at this issue sooner rather than later

Sgs moved this task from Inbox to Tracking on the Growth-Team board.Apr 2 2026, 2:09 PM
cjming added a comment.Apr 2 2026, 3:43 PM

thanks @jnuche for the heads up - duly noted - we will investigate

cjming mentioned this in T422209: Resolve ConfigsFetcher::VERSION updates during train deployments.Apr 2 2026, 8:14 PM
• Scott_French updated the task description. (Show Details)Apr 3 2026, 3:36 PM
• Scott_French updated the task description. (Show Details)Apr 3 2026, 3:41 PM
• Scott_French updated the task description. (Show Details)Apr 3 2026, 3:47 PM
Sfaci edited projects, added: Test Kitchen (Test Kitchen (Experiment Platform Sprint 22)); removed: Test Kitchen.Apr 6 2026, 9:24 AM
dancy subscribed.Apr 6 2026, 4:09 PM

I'm on train duty this week and these warnings have made themselves very noticeable, eclipsing other mediawiki log messages due to the volume when it happens.

There are two additional warnings that also happen:

Error 1
  • mwversion: 1.46.0-wmf.22
  • timestamp: 2026-04-06T15:55:15.535Z
  • phpversion: 8.3.30
  • reqId: 663e60f8-aff6-4698-a70e-5b15de8213de
  • Find reqId in Logstash
normalized_message
[{reqId}] {exception_url}   PHP Warning: Undefined array key "mobile-toc-abc2"
FrameLocationCall
from/srv/mediawiki/php-1.46.0-wmf.22/extensions/TestKitchen/includes/Sdk/ExperimentManager.php(207)
#0/srv/mediawiki/php-1.46.0-wmf.22/extensions/TestKitchen/includes/Sdk/ExperimentManager.php(207)MediaWiki\Exception\MWExceptionHandler::handleError(int, string, string, int)
#1/srv/mediawiki/php-1.46.0-wmf.22/extensions/TestKitchen/includes/Sdk/ExperimentManager.php(152)MediaWiki\Extension\TestKitchen\Sdk\ExperimentManager->getExperimentConfig(string, array)
#2/srv/mediawiki/php-1.46.0-wmf.22/extensions/ReaderExperiments/src/Hooks.php(74)MediaWiki\Extension\TestKitchen\Sdk\ExperimentManager->getExperiment(string)
#3/srv/mediawiki/php-1.46.0-wmf.22/extensions/ReaderExperiments/src/Hooks.php(393)MediaWiki\Extension\ReaderExperiments\Hooks->getAssignedGroup(MediaWiki\Request\WebRequest, string)
#4/srv/mediawiki/php-1.46.0-wmf.22/includes/HookContainer/HookContainer.php(135)MediaWiki\Extension\ReaderExperiments\Hooks->onShouldUseParsoid(MediaWiki\User\User, MediaWiki\Request\WebRequest, MediaWiki\Title\Title, bool)
#5/srv/mediawiki/php-1.46.0-wmf.22/extensions/ParserMigration/src/Hook/HookRunner.php(23)MediaWiki\HookContainer\HookContainer->run(string, array)
#6/srv/mediawiki/php-1.46.0-wmf.22/extensions/ParserMigration/src/Oracle.php(63)MediaWiki\Extension\ParserMigration\Hook\HookRunner->onShouldUseParsoid(MediaWiki\User\User, MediaWiki\Request\WebRequest, MediaWiki\Title\Title, bool)
#7/srv/mediawiki/php-1.46.0-wmf.22/extensions/ParserMigration/src/Hooks.php(195)MediaWiki\Extension\ParserMigration\Oracle->shouldUseParsoid(MediaWiki\User\User, MediaWiki\Request\WebRequest, MediaWiki\Title\Title)
#8/srv/mediawiki/php-1.46.0-wmf.22/includes/HookContainer/HookContainer.php(135)MediaWiki\Extension\ParserMigration\Hooks->onSidebarBeforeOutput(MediaWiki\Minerva\Skins\SkinMinerva, array)
#9/srv/mediawiki/php-1.46.0-wmf.22/includes/HookContainer/HookRunner.php(3926)MediaWiki\HookContainer\HookContainer->run(string, array, array)
#10/srv/mediawiki/php-1.46.0-wmf.22/includes/Skin/Skin.php(1608)MediaWiki\HookContainer\HookRunner->onSidebarBeforeOutput(MediaWiki\Minerva\Skins\SkinMinerva, array)
#11/srv/mediawiki/php-1.46.0-wmf.22/includes/Skin/SkinTemplate.php(633)MediaWiki\Skin\Skin->buildSidebar()
#12/srv/mediawiki/php-1.46.0-wmf.22/includes/Skin/SkinTemplate.php(179)MediaWiki\Skin\SkinTemplate->getPortletsTemplateData()
#13/srv/mediawiki/php-1.46.0-wmf.22/includes/Skin/SkinMustache.php(87)MediaWiki\Skin\SkinTemplate->getTemplateData()
#14/srv/mediawiki/php-1.46.0-wmf.22/skins/MinervaNeue/includes/Skins/SkinMinerva.php(227)MediaWiki\Skin\SkinMustache->getTemplateData()
#15/srv/mediawiki/php-1.46.0-wmf.22/includes/Skin/SkinMustache.php(51)MediaWiki\Minerva\Skins\SkinMinerva->getTemplateData()
#16/srv/mediawiki/php-1.46.0-wmf.22/includes/Skin/SkinTemplate.php(172)MediaWiki\Skin\SkinMustache->generateHTML()
#17/srv/mediawiki/php-1.46.0-wmf.22/includes/Skin/Skin.php(671)MediaWiki\Skin\SkinTemplate->outputPage()
#18/srv/mediawiki/php-1.46.0-wmf.22/includes/Output/OutputPage.php(3313)MediaWiki\Skin\Skin->outputPageFinal(MediaWiki\Output\OutputPage)
#19/srv/mediawiki/php-1.46.0-wmf.22/includes/Actions/ActionEntryPoint.php(162)MediaWiki\Output\OutputPage->output(bool)
#20/srv/mediawiki/php-1.46.0-wmf.22/includes/MediaWikiEntryPoint.php(180)MediaWiki\Actions\ActionEntryPoint->execute()
#21/srv/mediawiki/php-1.46.0-wmf.22/index.php(44)MediaWiki\MediaWikiEntryPoint->run()
#22/srv/mediawiki/w/index.php(3)require(string)
#23{main}
Error 2
  • mwversion: 1.46.0-wmf.22
  • timestamp: 2026-04-06T15:55:09.088Z
  • phpversion: 8.3.30
  • reqId: b0bc4483-8445-4454-8b66-cfbfecf096bd
  • Find reqId in Logstash
normalized_message
[{reqId}] {exception_url}   PHP Warning: Undefined array key "we-1-8-mobile-account-menu"
FrameLocationCall
from/srv/mediawiki/php-1.46.0-wmf.22/extensions/TestKitchen/includes/Sdk/ExperimentManager.php(207)
#0/srv/mediawiki/php-1.46.0-wmf.22/extensions/TestKitchen/includes/Sdk/ExperimentManager.php(207)MediaWiki\Exception\MWExceptionHandler::handleError(int, string, string, int)
#1/srv/mediawiki/php-1.46.0-wmf.22/extensions/TestKitchen/includes/Sdk/ExperimentManager.php(152)MediaWiki\Extension\TestKitchen\Sdk\ExperimentManager->getExperimentConfig(string, array)
#2/srv/mediawiki/php-1.46.0-wmf.22/skins/MinervaNeue/includes/Skins/SkinMinerva.php(450)MediaWiki\Extension\TestKitchen\Sdk\ExperimentManager->getExperiment(string)
#3/srv/mediawiki/php-1.46.0-wmf.22/skins/MinervaNeue/includes/Skins/SkinMinerva.php(471)MediaWiki\Minerva\Skins\SkinMinerva->shouldShowAccountMenuItems()
#4/srv/mediawiki/php-1.46.0-wmf.22/skins/MinervaNeue/includes/Skins/SkinMinerva.php(309)MediaWiki\Minerva\Skins\SkinMinerva->getPersonalToolsMenu(array)
#5/srv/mediawiki/php-1.46.0-wmf.22/includes/Skin/SkinMustache.php(51)MediaWiki\Minerva\Skins\SkinMinerva->getTemplateData()
#6/srv/mediawiki/php-1.46.0-wmf.22/includes/Skin/SkinTemplate.php(172)MediaWiki\Skin\SkinMustache->generateHTML()
#7/srv/mediawiki/php-1.46.0-wmf.22/includes/Skin/Skin.php(671)MediaWiki\Skin\SkinTemplate->outputPage()
#8/srv/mediawiki/php-1.46.0-wmf.22/includes/Output/OutputPage.php(3313)MediaWiki\Skin\Skin->outputPageFinal(MediaWiki\Output\OutputPage)
#9/srv/mediawiki/php-1.46.0-wmf.22/includes/Actions/ActionEntryPoint.php(162)MediaWiki\Output\OutputPage->output(bool)
#10/srv/mediawiki/php-1.46.0-wmf.22/includes/MediaWikiEntryPoint.php(180)MediaWiki\Actions\ActionEntryPoint->execute()
#11/srv/mediawiki/php-1.46.0-wmf.22/index.php(44)MediaWiki\MediaWikiEntryPoint->run()
#12/srv/mediawiki/w/index.php(3)require(string)
#13{main}
HakanIST subscribed.Apr 6 2026, 8:10 PM

Hi, I tried to investigate through the code to understand the warning. Two things came up;

  1. In the constructor (line ~50), enrollmentResult is initialized as []:
$this->enrollmentResult = [];

But getExperimentConfig() (line ~197) expects it to have overrides, assigned, and subject_ids keys:

$isOverridden = in_array( $experimentName, $this->enrollmentResult['overrides'] );
  1. In getExperiment(), when $experiments is empty (e.g. cold cache after ConfigsFetcher::VERSION bump), a cookie-enrolled user can pass the in_array check at line ~170 and reach getExperimentConfig() with missing data.
gerritbot added a comment.Apr 7 2026, 10:25 AM

Change #1268526 had a related patch set uploaded (by Phuedx; author: Phuedx):

[mediawiki/extensions/TestKitchen@master] PHP SDK: Handle experiment config missing or malformed

https://gerrit.wikimedia.org/r/1268526

gerritbot added a project: Patch-For-Review.Apr 7 2026, 10:25 AM
phuedx claimed this task.Apr 7 2026, 1:35 PM
phuedx moved this task from Ready for Development to In Progress on the Test Kitchen (Test Kitchen (Experiment Platform Sprint 22)) board.
gerritbot added a comment.Apr 7 2026, 7:42 PM

Change #1268526 merged by jenkins-bot:

[mediawiki/extensions/TestKitchen@master] PHP SDK: Handle experiment config missing or malformed

https://gerrit.wikimedia.org/r/1268526

ReleaseTaggerBot added a project: MW-1.46-notes (1.46.0-wmf.24; 2026-04-14).Apr 7 2026, 8:00 PM
gerritbot added a comment.Apr 7 2026, 8:04 PM

Change #1268651 had a related patch set uploaded (by Santiago Faci; author: Phuedx):

[mediawiki/extensions/TestKitchen@wmf/1.46.0-wmf.22] PHP SDK: Handle experiment config missing or malformed

https://gerrit.wikimedia.org/r/1268651

gerritbot added a comment.Apr 7 2026, 8:18 PM

Change #1268654 had a related patch set uploaded (by Santiago Faci; author: Phuedx):

[mediawiki/extensions/TestKitchen@wmf/1.46.0-wmf.23] PHP SDK: Handle experiment config missing or malformed

https://gerrit.wikimedia.org/r/1268654

gerritbot added a comment.Apr 7 2026, 9:08 PM

Change #1268651 merged by jenkins-bot:

[mediawiki/extensions/TestKitchen@wmf/1.46.0-wmf.22] PHP SDK: Handle experiment config missing or malformed

https://gerrit.wikimedia.org/r/1268651

gerritbot added a comment.Apr 7 2026, 9:21 PM

Change #1268654 merged by jenkins-bot:

[mediawiki/extensions/TestKitchen@wmf/1.46.0-wmf.23] PHP SDK: Handle experiment config missing or malformed

https://gerrit.wikimedia.org/r/1268654

Stashbot added a comment.Apr 7 2026, 9:31 PM

Mentioned in SAL (#wikimedia-operations) [2026-04-07T21:31:06Z] <cscott@deploy1003> Started scap sync-world: Backport for [[gerrit:1268654|PHP SDK: Handle experiment config missing or malformed (T422112)]], [[gerrit:1268634|ForeignWikiRequest: Pass session to internal 'centralauthtoken' request (T422218)]], [[gerrit:1268653|Remove Navigation Menu Link Instrumentation on Personal Dashboard (T422512)]], [[gerrit:1268648|Bump wikimedia/parsoid to 0.23.0-a26 (T422394)]], [[gerrit:12686

Stashbot mentioned this in T422394: CTT tasks week of 2026-04-03.Apr 7 2026, 9:31 PM
Stashbot mentioned this in T422512: Remove Navigation Menu Link Instrumentation.
Stashbot mentioned this in T422218: Marking cross-wiki notifications as read doesn't work.
Maintenance_bot removed a project: Patch-For-Review.Apr 7 2026, 9:31 PM
Stashbot added a comment.Apr 7 2026, 9:32 PM

Mentioned in SAL (#wikimedia-operations) [2026-04-07T21:32:55Z] <cscott@deploy1003> matmarex, sfaci, cscott, kgraessle: Backport for [[gerrit:1268654|PHP SDK: Handle experiment config missing or malformed (T422112)]], [[gerrit:1268634|ForeignWikiRequest: Pass session to internal 'centralauthtoken' request (T422218)]], [[gerrit:1268653|Remove Navigation Menu Link Instrumentation on Personal Dashboard (T422512)]], [[gerrit:1268648|Bump wikimedia/parsoid to 0.23.0-a26 (T422394)]], [[g

Stashbot added a comment.Apr 7 2026, 9:39 PM

Mentioned in SAL (#wikimedia-operations) [2026-04-07T21:39:25Z] <cscott@deploy1003> Finished scap sync-world: Backport for [[gerrit:1268654|PHP SDK: Handle experiment config missing or malformed (T422112)]], [[gerrit:1268634|ForeignWikiRequest: Pass session to internal 'centralauthtoken' request (T422218)]], [[gerrit:1268653|Remove Navigation Menu Link Instrumentation on Personal Dashboard (T422512)]], [[gerrit:1268648|Bump wikimedia/parsoid to 0.23.0-a26 (T422394)]], [[gerrit:1268

ReleaseTaggerBot edited projects, added: MW-1.46-notes (1.46.0-wmf.23; 2026-04-07); removed: MW-1.46-notes (1.46.0-wmf.24; 2026-04-14).Apr 7 2026, 10:00 PM
gerritbot added a comment.Apr 8 2026, 1:43 AM

Change #1268701 had a related patch set uploaded (by Phuedx; author: Phuedx):

[mediawiki/extensions/TestKitchen@master] PHP SDK: Measure known experiments correctly

https://gerrit.wikimedia.org/r/1268701

gerritbot added a project: Patch-For-Review.Apr 8 2026, 1:43 AM
phuedx added a comment.Apr 8 2026, 6:06 AM

I don't wish to speak too soon but I think we've seen the last of both warnings that @dancy mentioned above: https://logstash.wikimedia.org/goto/ee3aee2cf67d6b96eb7274a3899aecb9

phuedx added a comment.Apr 8 2026, 6:08 AM
In T422112#11791104, @HakanIST wrote:

Hi, I tried to investigate through the code to understand the warning. Two things came up;

  1. In the constructor (line ~50), enrollmentResult is initialized as []:
$this->enrollmentResult = [];

But getExperimentConfig() (line ~197) expects it to have overrides, assigned, and subject_ids keys:

$isOverridden = in_array( $experimentName, $this->enrollmentResult['overrides'] );
  1. In getExperiment(), when $experiments is empty (e.g. cold cache after ConfigsFetcher::VERSION bump), a cookie-enrolled user can pass the in_array check at line ~170 and reach getExperimentConfig() with missing data.

Yup. Good spots. There were a couple of accesses of $this->enrollmentResult and/or $experimentConfigs which could well have been empty.

gerritbot added a comment.Apr 8 2026, 7:48 AM

Change #1268701 merged by jenkins-bot:

[mediawiki/extensions/TestKitchen@master] PHP SDK: Measure known experiments correctly

https://gerrit.wikimedia.org/r/1268701

ReleaseTaggerBot edited projects, added: MW-1.46-notes (1.46.0-wmf.24; 2026-04-14); removed: MW-1.46-notes (1.46.0-wmf.23; 2026-04-07).Apr 8 2026, 8:00 AM
Maintenance_bot removed a project: Patch-For-Review.Apr 8 2026, 8:32 AM
gerritbot added a comment.Apr 8 2026, 1:09 PM

Change #1268947 had a related patch set uploaded (by Phuedx; author: Phuedx):

[mediawiki/extensions/TestKitchen@wmf/1.46.0-wmf.22] PHP SDK: Measure known experiments correctly

https://gerrit.wikimedia.org/r/1268947

gerritbot added a project: Patch-For-Review.Apr 8 2026, 1:09 PM

Change #1268948 had a related patch set uploaded (by Phuedx; author: Phuedx):

[mediawiki/extensions/TestKitchen@wmf/1.46.0-wmf.23] PHP SDK: Measure known experiments correctly

https://gerrit.wikimedia.org/r/1268948

gerritbot added a comment.Apr 8 2026, 1:32 PM

Change #1268948 merged by jenkins-bot:

[mediawiki/extensions/TestKitchen@wmf/1.46.0-wmf.23] PHP SDK: Measure known experiments correctly

https://gerrit.wikimedia.org/r/1268948

gerritbot added a comment.Apr 8 2026, 1:32 PM

Change #1268947 merged by jenkins-bot:

[mediawiki/extensions/TestKitchen@wmf/1.46.0-wmf.22] PHP SDK: Measure known experiments correctly

https://gerrit.wikimedia.org/r/1268947

Stashbot added a comment.Apr 8 2026, 1:33 PM

Mentioned in SAL (#wikimedia-operations) [2026-04-08T13:33:22Z] <phuedx@deploy1003> Started scap sync-world: Backport for [[gerrit:1268948|PHP SDK: Measure known experiments correctly (T422112)]], [[gerrit:1268947|PHP SDK: Measure known experiments correctly (T422112)]]

Stashbot added a comment.Apr 8 2026, 1:35 PM

Mentioned in SAL (#wikimedia-operations) [2026-04-08T13:35:16Z] <phuedx@deploy1003> phuedx: Backport for [[gerrit:1268948|PHP SDK: Measure known experiments correctly (T422112)]], [[gerrit:1268947|PHP SDK: Measure known experiments correctly (T422112)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.

Stashbot added a comment.Apr 8 2026, 1:41 PM

Mentioned in SAL (#wikimedia-operations) [2026-04-08T13:41:21Z] <phuedx@deploy1003> Finished scap sync-world: Backport for [[gerrit:1268948|PHP SDK: Measure known experiments correctly (T422112)]], [[gerrit:1268947|PHP SDK: Measure known experiments correctly (T422112)]] (duration: 07m 58s)

ReleaseTaggerBot edited projects, added: MW-1.46-notes (1.46.0-wmf.22; 2026-03-31); removed: MW-1.46-notes (1.46.0-wmf.24; 2026-04-14).Apr 8 2026, 2:00 PM
Maintenance_bot removed a project: Patch-For-Review.Apr 8 2026, 2:32 PM
gerritbot added a comment.Apr 13 2026, 9:14 AM

Change #1270371 had a related patch set uploaded (by Phuedx; author: Phuedx):

[mediawiki/extensions/TestKitchen@master] ConfigsFetcher: Track cache/stash hit rate

https://gerrit.wikimedia.org/r/1270371

gerritbot added a project: Patch-For-Review.Apr 13 2026, 9:14 AM
gerritbot added a comment.Apr 13 2026, 12:34 PM

Change #1270371 merged by jenkins-bot:

[mediawiki/extensions/TestKitchen@master] ConfigsFetcher: Track cache/stash hit rate

https://gerrit.wikimedia.org/r/1270371

ReleaseTaggerBot edited projects, added: MW-1.46-notes (1.46.0-wmf.24; 2026-04-14); removed: MW-1.46-notes (1.46.0-wmf.22; 2026-03-31).Apr 13 2026, 1:00 PM
Maintenance_bot removed a project: Patch-For-Review.Apr 13 2026, 1:31 PM
gerritbot added a comment.Apr 17 2026, 3:09 PM

Change #1273841 had a related patch set uploaded (by Phuedx; author: Phuedx):

[mediawiki/extensions/TestKitchen@master] PHP SDK: Split measurement of unknown experiments

https://gerrit.wikimedia.org/r/1273841

gerritbot added a project: Patch-For-Review.Apr 17 2026, 3:09 PM
gerritbot added a comment.Apr 20 2026, 9:30 AM

Change #1273841 merged by jenkins-bot:

[mediawiki/extensions/TestKitchen@master] PHP SDK: Split measurement of unknown experiments

https://gerrit.wikimedia.org/r/1273841

Maintenance_bot removed a project: Patch-For-Review.Apr 20 2026, 9:34 AM
gerritbot added a comment.Apr 20 2026, 9:35 AM

Change #1275357 had a related patch set uploaded (by Phuedx; author: Phuedx):

[mediawiki/extensions/TestKitchen@wmf/1.46.0-wmf.24] PHP SDK: Split measurement of unknown experiments

https://gerrit.wikimedia.org/r/1275357

gerritbot added a project: Patch-For-Review.Apr 20 2026, 9:35 AM
ReleaseTaggerBot edited projects, added: MW-1.46-notes (1.46.0-wmf.26; 2026-04-28); removed: MW-1.46-notes (1.46.0-wmf.24; 2026-04-14).Apr 20 2026, 10:00 AM
gerritbot added a comment.Apr 20 2026, 1:21 PM

Change #1275357 merged by jenkins-bot:

[mediawiki/extensions/TestKitchen@wmf/1.46.0-wmf.24] PHP SDK: Split measurement of unknown experiments

https://gerrit.wikimedia.org/r/1275357

Stashbot added a comment.Apr 20 2026, 1:22 PM

Mentioned in SAL (#wikimedia-operations) [2026-04-20T13:22:05Z] <phuedx@deploy1003> Started scap sync-world: Backport for [[gerrit:1275357|PHP SDK: Split measurement of unknown experiments (T422112)]]

Stashbot added a comment.Apr 20 2026, 1:23 PM

Mentioned in SAL (#wikimedia-operations) [2026-04-20T13:23:41Z] <phuedx@deploy1003> phuedx: Backport for [[gerrit:1275357|PHP SDK: Split measurement of unknown experiments (T422112)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.

Stashbot added a comment.Apr 20 2026, 1:29 PM

Mentioned in SAL (#wikimedia-operations) [2026-04-20T13:29:56Z] <phuedx@deploy1003> Finished scap sync-world: Backport for [[gerrit:1275357|PHP SDK: Split measurement of unknown experiments (T422112)]] (duration: 07m 51s)

Maintenance_bot removed a project: Patch-For-Review.Apr 20 2026, 1:31 PM
ReleaseTaggerBot edited projects, added: MW-1.46-notes (1.46.0-wmf.24; 2026-04-14); removed: MW-1.46-notes (1.46.0-wmf.26; 2026-04-28).Apr 20 2026, 2:00 PM
KReid-WMF edited projects, added: Test Kitchen (Experiment Platform Sprint 23); removed: Test Kitchen (Test Kitchen (Experiment Platform Sprint 22)).Apr 23 2026, 3:15 PM
KReid-WMF moved this task from Ready for Development to In Progress on the Test Kitchen (Experiment Platform Sprint 23) board.
phuedx closed this task as Resolved.Apr 27 2026, 11:12 AM

I believe that this is Fixed Enough™ to be closed as Resolved. The reason for my trepidation here is that we didn't have enough signals about the system to understand its behaviour at the time, which is… ironic. However, I feel confident in saying the following:

  1. There are two flaws in the design of the cache/stash update strategy:
    1. An hard-coded version number is added to the key used to fetch experiment and instrument configs so but the maintenance script that does the updating is run against aawiki, a group 2 wiki. In the event of a version bump, the maintenance script will continue to store experiment and instrument configs against with the old version number while group 0 and group 1 wikis are fetching using the new version number. This will continue until group 2 wikis are up to date
    2. We only invalidate the cache if the value of the stash is out of date. Therefore, it's possible that an invalid/out of date value could get stuck in the cache as long as the value in the stash is up to date (e.g. a new experiment is configured in Test Kitchen UI). If the maintenance script doesn't invalidate the cache, it should be updated on a cache miss followed by a read from the stash.
  2. The original patch introduced a bug wherein if a user was enrolled in an experiment, we wouldn't check if we had an experiment config before attempting to access it, triggering a warning. You can see how this is exacerbated by (1.A). For example: If an everyone experiment is active and we bump the version, then requests to group0 and group1 wikis can include enrollment information for an experiment that the appservers don't know exists.

Fixing (1.A) and (1.B) will be done as part of a follow-on task (see Actions below). (2) has been fixed as part of this task.

As well as fixing (2), we've added a whole bunch of metrics covering both how \MediaWiki\Extension\TestKitchen\Sdk\ExperimentManager#getExperiment() and \MediaWiki\Extension\ConfigFetcher#getConfigsInternal() are functioning. These metrics include:

  • The number of calls to ExperimentManager#getExperiment() by experiment
    • This includes calls where where the experiment is known/unknown – we have/don't have an experiment config respectively
  • The ConfigsFetcher#getConfigsInternal() cache/stash hit/miss rates

We have the beginnings of a dashboard based on these metrics here, which we'll be iterating on. The cache hit rate is very nearly almost 100%, which is great, but it means that it's unlikely that the cache will be invalidated (see 1.B. above)

Actions

  1. Update the maintenance script to run against group0, group1, and group2 wikis. We should do this before we start working on SDS 2.3 in ernest as it will likely involve a version bump
  2. Update the maintenance script to invalidate the cache and the stash independently
  3. Introduce a tunable, probabalistic cache miss mechanism in ConfigsFetcher#getConfigsInternal(), which causes the cache miss codepath to execute 1 in every 1000 calls to ensure that both the cache is being invalidated and that the stash codepath is being executed
  4. Test how another version bump affects the system at large. I'm confident that there will be no flood of errors and that the metrics that we're collecting will highlight any other issues in the design of the system
phuedx moved this task from In Progress to Done on the Test Kitchen (Experiment Platform Sprint 23) board.Apr 27 2026, 11:12 AM
phuedx mentioned this in T424513: Make ConfigsFetcher invalidate the cache and stash independently.Apr 27 2026, 12:14 PM
phuedx added a parent task: T424513: Make ConfigsFetcher invalidate the cache and stash independently.
phuedx added a parent task: T422209: Resolve ConfigsFetcher::VERSION updates during train deployments.
phuedx added a comment.Apr 27 2026, 2:03 PM
In T422112#11860469, @phuedx wrote:

Actions

  1. Update the maintenance script to run against group0, group1, and group2 wikis. We should do this before we start working on SDS 2.3 in ernest as it will likely involve a version bump

T422209: Resolve ConfigsFetcher::VERSION updates during train deployments

  1. Update the maintenance script to invalidate the cache and the stash independently

T424513: Make ConfigsFetcher invalidate the cache and stash independently

  1. Introduce a tunable, probabalistic cache miss mechanism in ConfigsFetcher#getConfigsInternal(), which causes the cache miss codepath to execute 1 in every 1000 calls to ensure that both the cache is being invalidated and that the stash codepath is being executed

I think we could hold off on this until after T424513 is deployed and we've tested a version bump in production.

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits