We want to use [[ https://wikitech.wikimedia.org/wiki/Test_Kitchen | Test Kitchen ]] for future A/B testing. Let's make an experimental change that is controlled through this framework, and learn about what is involved.
- [x] Read technical documentation and examples, especially https://wikitech.wikimedia.org/wiki/Test_Kitchen/Conduct_an_experiment
- [x] Try to record what we learn here in the task. --> see below
- [x] Write a docker-dev module to enable Test Kitchen (see https://wikitech.wikimedia.org/wiki/Test_Kitchen/Local_development_setup )
- Install and enable the MetricsPlatform extension
- Configure event streams
- Configure a local EventGate server
- [x] Code for review: https://gitlab.com/wmde/technical-wishes/docker-dev/-/merge_requests/188
- [x] Make a small but visible change to the reference dialog, for example changing the color of a button. {F71539189} https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Cite/+/1222589
- [x] Control this change through the test framework, using the [[ https://wikitech.wikimedia.org/wiki/Test_Kitchen/SDK/JavaScript_SDK | JavaScript SDK ]].
- [x] Set up instrumentation so that we know that people clicking the button were in condition A or B.
- [x] Set up event schema and streams; test both enrollment cases.
( We're not setting up a custom schema or stream for the test )
- [x] Deploy the change https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Cite/+/1222589
- [x] Create AB Test for *only* test.wikipedia.org https://test-kitchen.wikimedia.org/experiment/test-context-edit-color
- [x] Enroll the experiment for [[ https://wikitech.wikimedia.org/wiki/Test_Kitchen/Automated_analysis_of_experiments#Register_experiment | auto analysis ]]
https://gitlab.wikimedia.org/repos/product-analytics/test-kitchen/experiment-analytics-configs/-/merge_requests/43
- [] generate clicks on https://test.wikipedia.org/w/index.php?title=CiteDetailsTests&action=edit or other pages in the test wiki
- [] Look into the results of the experiment.
Results should be available in [[ https://superset.wikimedia.org/superset/dashboard/experiment-analytics/?native_filters_key=eKrzZO3XQT4_-rtxNKJ5Y5pYpoQr6A_LhkkQ62q7dPzmKAk3Egu6ExLRW_NytdVS | Superset ]]
##Documentation
AB Test for green button: https://test-kitchen.wikimedia.org/experiment/test-context-edit-color
Start Date: 2026-01-31
End Date: 2026-02-06
Test Wikipedia: 5 %
##For review
Our learnings:
[[ https://docs.google.com/document/d/1oVAEhBUaCS1plJuIXU6OAZSjfnnOoz4d-M1sZlMBr5Y/edit?usp=sharing | Test Kitchen Summary ]]
Test code change on beta:
http://dev.wiki.local.wmftest.net:8080/w/index.php?title=Example/Cite&veaction=edit&mpo=test-context-edit-color:green-button
To test https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Cite/+/1222589 locally:
# Start docker with eventlogging
# Add `?mpo=test-context-edit-color:green-button` to the url.
# Open a context item.
# Enjoy the light green icon. :)