• mwversion: 1.46.0-wmf.22
• timestamp: 2026-03-31T07:50:02.298Z
• phpversion: 8.3.30
• reqId: 2d0e0e52-7f9c-4a9b-8e06-cca6d99b99b4
• Find reqId in Logstash

[{reqId}] {exception_url}   PHP Warning: Undefined array key "stream_name"

• mwversion: 1.46.0-wmf.22
• timestamp: 2026-03-31T07:54:04.828Z
• phpversion: 8.3.30
• reqId: 95100eec-b69f-4dc3-8c79-e3d42a5124ae
• Find reqId in Logstash

[{reqId}] {exception_url}   PHP Warning: Undefined array key "contextual_attributes"

In the last hour just for the test wikis:

Flagged to (the rest of) Growth engineers

The generally-helpful changelog is now empty... Is that intentional?

The usage was recently modified (https://gerrit.wikimedia.org/r/c/mediawiki/extensions/TestKitchen/+/1251216). As far as I can see, this seems to be an issue in Test Kitchen internals rather than Growth's usage of it. CC @cjming @Sfaci @phuedx.

In T421828#11771118, @Urbanecm_WMF wrote:

The usage was recently modified (https://gerrit.wikimedia.org/r/c/mediawiki/extensions/TestKitchen/+/1251216). As far as I can see, this seems to be an issue in Test Kitchen internals rather than Growth's usage of it. CC @cjming @Sfaci @phuedx.

I think so. It's related to that change and also to other we made in Test Kitchen UI to add new fields to an API response that TestKitchen extension consumes (https://gitlab.wikimedia.org/repos/data-engineering/test-kitchen/-/merge_requests/309).
It seems that TestKitchen is not finding those new fields in the response and that's why we are seeing those errors about user_identifier_type, stream_name and contextual_attributes. The endpoint URL and the response that the extension consumes changed

Moving out of our sprint in favor of Experiment Platform; feel free to ping us for assistance.

For now, the only way I was able to reproduce exactly the same errors in my local was assuming that the response that comes from Test Kitchen API is the old one (where user_identifier_type, stream_name and contextual_attributes didn't exist).

And that probably doesn't make sense because the maintenance job is working fine according to their own logs (edited) 

Related to the switchover that was done last week? Maintenance jobs are running in a different cluster since then. I don't know if it was a coincidence but the maintenance job failed once last week two days after the switchover (because of timeout from etcd). But SREs considered that error as something not relevant.

Posting this here for some related context just in case. I don't think the current error is related to it

Related to the switchover that was done last week? Maintenance jobs are running in a different cluster since then

Silly question... If test-kitchen is running in both eqiad and codfw and the errors seem like the old response is returned, maybe we only deployed the new test-kitchen to codfw and not to eqiad? But I have no clue where testkitchen is deployed.

I don't know if it was a coincidence but the maintenance job failed once last week two days after the switchover (because of timeout from etcd). But SREs considered that error as something not relevant.

I agree. etcd timeouts are happening from time to time unfortunately. I wouldn't suspect the other error to have something in common with this one.

In T421828#11772338, @Urbanecm_WMF wrote:

Related to the switchover that was done last week? Maintenance jobs are running in a different cluster since then

Silly question... If test-kitchen is running in both eqiad and codfw and the errors seem like the old response is returned, maybe we only deployed the new test-kitchen to codfw and not to eqiad? But I have no clue where testkitchen is deployed.

test-kitchen is running in dse-k8s-eqiad kubernetes cluster which means, if I'm not understanding wrongly, that is only running in eqiad.

I agree. etcd timeouts are happening from time to time unfortunately. I wouldn't suspect the other error to have something in common with this one.

Right! In fact I have created a sample NOOP test in test-kitchen and I have confirmed that a production wiki was updated propertly with the enrollment for that one. That would mean that the maintenance job is updating things properly, at least in the wikis that are running wmf.21 where they are using the old endpoint

I also have a silly question: Could it happen that, somehow, the maintenance job was still running the code from wmf.21 instead the new branch wmf.22? That would explain why things are updated according to the old way but not according to the new one

Suggest we bump the CACHE_VERSION in ConfigFetchers

I had this issue locally before I updated my docker containers for test kitchen devserver.

There must be experiment configs in cache from before the change that added user_identifier_type.

Change #1265424 had a related patch set uploaded (by Santiago Faci; author: Santiago Faci):

[mediawiki/extensions/TestKitchen@master] ConfigsFetcher: Increasing the cache version

https://gerrit.wikimedia.org/r/1265424

Change #1265424 merged by jenkins-bot:

[mediawiki/extensions/TestKitchen@master] ConfigsFetcher: Increasing the cache version

https://gerrit.wikimedia.org/r/1265424

Change #1265436 had a related patch set uploaded (by Clare Ming; author: Santiago Faci):

[mediawiki/extensions/TestKitchen@wmf/1.46.0-wmf.22] ConfigsFetcher: Increasing the cache version

https://gerrit.wikimedia.org/r/1265436

Change #1265436 merged by jenkins-bot:

[mediawiki/extensions/TestKitchen@wmf/1.46.0-wmf.22] ConfigsFetcher: Increasing the cache version

https://gerrit.wikimedia.org/r/1265436

Mentioned in SAL (#wikimedia-operations) [2026-03-31T14:28:19Z] <cjming@deploy1003> Started scap sync-world: Backport for [[gerrit:1265436|ConfigsFetcher: Increasing the cache version (T421828)]]

Mentioned in SAL (#wikimedia-operations) [2026-03-31T14:30:15Z] <cjming@deploy1003> cjming: Backport for [[gerrit:1265436|ConfigsFetcher: Increasing the cache version (T421828)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.

Mentioned in SAL (#wikimedia-operations) [2026-03-31T14:36:31Z] <cjming@deploy1003> Finished scap sync-world: Backport for [[gerrit:1265436|ConfigsFetcher: Increasing the cache version (T421828)]] (duration: 08m 12s)

backported UBN fix to wmf.22 and it appears errors are dropping 😌

It seems that the schema for PHP configs have change to now have user_identifier_type as a required property (Coordination/EnrollmentsProcessor.php#19). fyi, I've updated TestKitchen#Testing_an_Experiment_on_Logged-In_Users accordingly for those who work with local configs instead of remote fetching. Hope that makes sense, also maybe there are other places to update that I've missed.

In T421828#11777127, @Sgs wrote:

It seems that the schema for PHP configs have change to now have user_identifier_type as a required property (Coordination/EnrollmentsProcessor.php#19). fyi, I've updated TestKitchen#Testing_an_Experiment_on_Logged-In_Users accordingly for those who work with local configs instead of remote fetching. Hope that makes sense, also maybe there are other places to update that I've missed.

Good catch! Documentation still needs to be updated. In fact there are also two name fields, stream_name and contextual_attributes (you will see related errors as comments in this task) and they will have to be added as well to that local config variable.
Anyway we will be working on updating documentation in the following days. There are some also some follow up work pending to do related to this although I think they won't change a lot how all this works right now
Thanks!