NOTE: SDS 2.4.20: If our documentation explains and shows how to use our suite of tools to conduct experiments, product teams (namely Web) will be able to learn how to instrument, launch, and evaluate their A/B tests themselves using our tools.
Acceptance criteria
- Documentation includes instructions and a full example for instrumenting an experiment using our new APIs/SDKs such that a product team has enough information to get started with conducting an experiment using Experimentation Lab
- Using CSS body classes (implemented in T393101: Experiment developer experience: automatic CSS tags) for testing appearance changes
- Gating PHP code behind experiment group assignment
- Gating JS code behind experiment group assignment
- Collecting interaction data for experiments – producing events in JS, and noting the PHP counterpart
- Using CTR instrument (depends on T394675) and T394841: [Automated Analytics] Generic CTR metric definitions to measure engagement
- Configuring experiment (manual for now, will be doable with xLab UI after T392911 and other tasks)
Implementation notes
Improve:
- https://wikitech.wikimedia.org/wiki/Metrics_Platform/Conduct_an_experiment#Instrumentation_spec
- https://wikitech.wikimedia.org/wiki/Metrics_Platform/Conduct_an_experiment#Code
- https://wikitech.wikimedia.org/wiki/Metrics_Platform/Conduct_an_experiment#Launch
- https://wikitech.wikimedia.org/wiki/Metrics_Platform/Conduct_an_experiment#Example
Some considerations:
Provide CSS and Javascript guidance that prevents flashing for end users (define treatment name in CSS classes in the string variables)
tell the experiment owner that a logged in user may fall out of the experiment
References:
- Sam/Julie/Mikhail session 1 recording (Google Drive)
- Sam/Julie/Mikhail session 2 recording (Google Drive)