We want to use Test Kitchen for future A/B testing. Let's make an experimental change that is controlled through this framework, and learn about what is involved.

• Read technical documentation and examples, especially https://wikitech.wikimedia.org/wiki/Test_Kitchen/Conduct_an_experiment
• Try to record what we learn here in the task. --> see below
• Write a docker-dev module to enable Test Kitchen (see https://wikitech.wikimedia.org/wiki/Test_Kitchen/Local_development_setup )
  • Install and enable the MetricsPlatform extension
  • Configure event streams
  • Configure a local EventGate server
  • Code for review: https://gitlab.com/wmde/technical-wishes/docker-dev/-/merge_requests/188
• Make a small but visible change to the reference dialog, for example changing the color of a button.
  
  https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Cite/+/1222589
• Control this change through the test framework, using the JavaScript SDK.
• Set up instrumentation so that we know that people clicking the button were in condition A or B.
• Set up event schema and streams; test both enrollment cases. ( We're not setting up a custom schema or stream for the test )
• Deploy the change https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Cite/+/1222589
• Create AB Test for *only* test.wikipedia.org https://test-kitchen.wikimedia.org/experiment/test-context-edit-color
• Enroll the experiment for auto analysis https://gitlab.wikimedia.org/repos/product-analytics/test-kitchen/experiment-analytics-configs/-/merge_requests/43
• Generate clicks on https://test.wikipedia.org/w/index.php?title=CiteDetailsTests&action=edit or other pages in the test wiki
  
• Look into the results of the experiment. Results should be available in Superset
  

Documentation

AB Test for green button: https://test-kitchen.wikimedia.org/experiment/test-context-edit-color
Start Date: 2026-01-31
End Date: 2026-02-06
Test Wikipedia: 5 %

For review

Our learnings:
Test Kitchen Summary

Test code change on beta:
http://dev.wiki.local.wmftest.net:8080/w/index.php?title=Example/Cite&veaction=edit&mpo=test-context-edit-color:green-button

To test https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Cite/+/1222589 locally:

1. Start docker with eventlogging
2. Add ?mpo=test-context-edit-color:green-button to the url.
3. Open a context item.
4. Enjoy the light green icon. :)