Page MenuHomePhabricator
Create Task
Maniphest T414528

Run synthetic experiment using Javascript SDK in Test Kitchen
Open, HighPublic1 Estimated Story Points
Actions

Description

Background

To verify that the switch to Test Kitchen is working as expected (see T407806: Rename Metrics Platform Extension to Test Kitchen), run some experiments using the Javascript SDK in Test Kitchen.

Description

Replicate previous experiments to ensure the Test Kitchen Javascript SDK is working properly.

Previous experiments:

Acceptance Criteria

  • Expected results are confirmed in Hive
  • Instrumentation is removed upon verification

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
Remove experiment code testing TK SDKsmediawiki/extensions/TestKitchenmaster+2 -84
Update experiment code for JS, PHP SDKs testing of TKmediawiki/extensions/TestKitchenwmf/1.46.0-wmf.11+50 -220
Update experiment code for JS, PHP SDKs testing of TKmediawiki/extensions/TestKitchenmaster+50 -220
Customize query in gerrit

Related Objects

Event Timeline

cjming created this task.Jan 13 2026, 8:11 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 13 2026, 8:11 PM
cjming renamed this task from Run synthetic experiment using Javascript SDK using Test Kitchen to Run synthetic experiment using Javascript SDK with Test Kitchen.Jan 13 2026, 8:12 PM
cjming renamed this task from Run synthetic experiment using Javascript SDK with Test Kitchen to Run synthetic experiment using Javascript SDK in Test Kitchen.Jan 14 2026, 7:35 AM
cjming mentioned this in T407806: Rename Metrics Platform Extension to Test Kitchen.Thu, Jan 15, 4:02 AM
gerritbot added a comment.Thu, Jan 15, 8:17 PM

Change #1227426 had a related patch set uploaded (by Clare Ming; author: Clare Ming):

[mediawiki/extensions/TestKitchen@master] Update experiment code for JS, PHP SDKs testing of TK

https://gerrit.wikimedia.org/r/1227426

gerritbot added a project: Patch-For-Review.Thu, Jan 15, 8:17 PM
gerritbot added a comment.Thu, Jan 15, 8:58 PM

Change #1227426 merged by jenkins-bot:

[mediawiki/extensions/TestKitchen@master] Update experiment code for JS, PHP SDKs testing of TK

https://gerrit.wikimedia.org/r/1227426

gerritbot added a comment.Thu, Jan 15, 8:59 PM

Change #1227435 had a related patch set uploaded (by Clare Ming; author: Clare Ming):

[mediawiki/extensions/TestKitchen@wmf/1.46.0-wmf.11] Update experiment code for JS, PHP SDKs testing of TK

https://gerrit.wikimedia.org/r/1227435

gerritbot added a comment.Thu, Jan 15, 9:52 PM

Change #1227435 merged by jenkins-bot:

[mediawiki/extensions/TestKitchen@wmf/1.46.0-wmf.11] Update experiment code for JS, PHP SDKs testing of TK

https://gerrit.wikimedia.org/r/1227435

Stashbot added a comment.Thu, Jan 15, 9:53 PM

Mentioned in SAL (#wikimedia-operations) [2026-01-15T21:53:05Z] <cjming@deploy2002> Started scap sync-world: Backport for [[gerrit:1227435|Update experiment code for JS, PHP SDKs testing of TK (T414528 T414530)]]

Stashbot mentioned this in T414530: Run synthetic experiment using PHP SDK in Test Kitchen.Thu, Jan 15, 9:53 PM

Mentioned in SAL (#wikimedia-operations) [2026-01-15T21:55:07Z] <cjming@deploy2002> cjming: Backport for [[gerrit:1227435|Update experiment code for JS, PHP SDKs testing of TK (T414528 T414530)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.

Stashbot added a comment.Thu, Jan 15, 10:02 PM

Mentioned in SAL (#wikimedia-operations) [2026-01-15T22:02:29Z] <cjming@deploy2002> Finished scap sync-world: Backport for [[gerrit:1227435|Update experiment code for JS, PHP SDKs testing of TK (T414528 T414530)]] (duration: 09m 23s)

Maintenance_bot removed a project: Patch-For-Review.Thu, Jan 15, 10:31 PM
cjming claimed this task.Fri, Jan 16, 7:06 PM
cjming triaged this task as High priority.
cjming moved this task from Incoming to Test Kitchen (Experiment Platform Sprint 18) on the Test Kitchen board.
cjming edited projects, added Test Kitchen (Test Kitchen (Experiment Platform Sprint 18)); removed Test Kitchen.
cjming moved this task from Ready for Development to To Deploy on the Test Kitchen (Test Kitchen (Experiment Platform Sprint 18)) board.
cjming set the point value for this task to 2.
gerritbot added a comment.Fri, Jan 16, 9:01 PM

Change #1227902 had a related patch set uploaded (by Clare Ming; author: Clare Ming):

[mediawiki/extensions/TestKitchen@master] Remove experiment code testing TK SDKs

https://gerrit.wikimedia.org/r/1227902

gerritbot added a project: Patch-For-Review.Fri, Jan 16, 9:01 PM
cjming added a comment.Fri, Jan 16, 9:07 PM

Process for testing Javascript SDK using Test Kitchen extension:

  1. Javascript experiment was configured and initiated yesterday on Japanese Wikipedia at 0.5% sampling
  2. Experiment code for both SDKs was backported to 1.46.0-wmf.11 yesterday
  3. Verified via Kafka topic streaming and running queries on analytics servers:

Confirmed that events were streaming in via:

kafkacat -C -b kafka-jumbo1010.eqiad.wmnet:9092 -t eqiad.product_metrics.web_base -o -100000 | grep TestKitchenMediaWikiModuleLoaded

Screenshot 2026-01-15 at 3.05.49 PM.png (460×3 px, 361 KB)

Ran the following queries:

SELECT count(*)
FROM event.product_metrics_web_base
WHERE instrument_name = 'TestKitchenMediaWikiModuleLoaded' AND year = 2026 AND month = 01 AND day = 16; 

// Returns 117035

SELECT count(*)
FROM event.product_metrics_web_base
WHERE instrument_name = 'TestKitchenMediaWikiModuleLoaded'
AND experiment.assigned='control'
AND year = 2026 AND month = 01 AND day = 16;

// Returns 59671

SELECT count(*)
FROM event.product_metrics_web_base
WHERE instrument_name = 'TestKitchenMediaWikiModuleLoaded'
AND experiment.assigned='treatment'
AND year = 2026 AND month = 01 AND day = 16;

// Returns 57364

With confirmations above, I will turn off experiment and remove related code.

Restricted Application added a subscriber: Dragoniez. · View Herald TranscriptFri, Jan 16, 9:07 PM
cjming updated the task description. (Show Details)Fri, Jan 16, 9:07 PM
cjming moved this task from To Deploy to Needs Review on the Test Kitchen (Test Kitchen (Experiment Platform Sprint 18)) board.
cjming changed the point value for this task from 2 to 1.
Sfaci moved this task from Needs Review to To Deploy on the Test Kitchen (Test Kitchen (Experiment Platform Sprint 18)) board.Fri, Jan 23, 12:58 AM
gerritbot added a comment.Fri, Jan 23, 1:20 AM

Change #1227902 merged by jenkins-bot:

[mediawiki/extensions/TestKitchen@master] Remove experiment code testing TK SDKs

https://gerrit.wikimedia.org/r/1227902

Maintenance_bot removed a project: Patch-For-Review.Fri, Jan 23, 1:31 AM
KReid-WMF edited projects, added Test Kitchen (Experiment Platform Sprint 19); removed Test Kitchen (Test Kitchen (Experiment Platform Sprint 18)).Thu, Jan 29, 5:02 PM
KReid-WMF moved this task from Ready for Development to To Deploy on the Test Kitchen (Experiment Platform Sprint 19) board.
phuedx moved this task from To Deploy to Done on the Test Kitchen (Experiment Platform Sprint 19) board.Mon, Feb 2, 4:09 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits