Change Details

NOTE: {T414729} includes basic instructions on usage when conducting experiments, this task is about more extensive documentation Once exposure logging is implemented, we should provide developers with more thorough guidance for logging exposure in their experiments to ensure accuracy and trustworthiness of the data collected. ## Acceptance criteria [ ] https://wikitech.wikimedia.org/wiki/Test_Kitchen/Conduct_an_experiment includes: - examples/scenarios - do's & don't's - recommended patterns / best practices

NOTE: {T414729} includes basic instructions on usage when conducting experiments, this task is about more extensive documentation Once exposure logging is implemented, we should provide developers with more thorough guidance for logging exposure in their experiments to ensure accuracy and trustworthiness of the data collected. ## Notes For everyone (all user traffic) experiments (that use edge uniques for enrollment) with server-side changes, we can borrow ideas from this approach: >>! In T414732#11570245, @mpopov wrote: > Marco from Reader Growth shared [[ https://github.com/wikimedia/mediawiki-extensions-ReaderExperiments/blob/043716c5a3050c702b552eb045c02d0cf9cb83c7/src/Hooks.php#L166 | their server-side implementation ]] with me. In summary: > - The code makes use of a helper function [[ https://github.com/wikimedia/mediawiki-extensions-ReaderExperiments/blob/043716c5a3050c702b552eb045c02d0cf9cb83c7/src/Hooks.php#L66 | isInAnyTreatmentGroup ]] which checks if the request is assigned to any group for a given experiment > - If the request is assigned to any group and satisfies other conditions (e.g. main namespace, skin), it loads `ext.readerExperiments.imageBrowsing` module and also adds `<div id="ext-readerExperiments-imageBrowsing"></div>` to the response. > - Therefore, all users enrolled in the experiment would have this client-side module loaded. > - The module then checks for enrollment/assignment and > - If user is in control group, do nothing to `div#ext-readerExperiments-imageBrowsing` > - If user is in treatment group, populate `div#ext-readerExperiments-imageBrowsing` > With this pattern, they could easily call `Experiment#logExposure()` on the client-side and have it be accurate (only log exposure when there is actually exposure to any variation). > > So they effectively have a solution to this task simply by only loading & running client-side code using server-side logic. As long as they continue to use this pattern for implementing their experiments, they're all good. We can provide a recommendation for how to log exposure on the client-side, namely by using server-side logic to conditionally load client-side instrumentation which would include an exposure logging call. ## Acceptance criteria [ ] https://wikitech.wikimedia.org/wiki/Test_Kitchen/Conduct_an_experiment includes: - examples/scenarios - do's & don't's - recommended patterns / best practices

NOTE: {T414729} includes basic instructions on usage when conducting experiments, this task is about more extensive documentation Once exposure logging is implemented, we should provide developers with more thorough guidance for logging exposure in their experiments to ensure accuracy and trustworthiness of the data collected. ## Notes For everyone (all user traffic) experiments (that use edge uniques for enrollment) with server-side changes, we can borrow ideas from this approach: >>! In T414732#11570245, @mpopov wrote: > Marco from Reader Growth shared [[ https://github.com/wikimedia/mediawiki-extensions-ReaderExperiments/blob/043716c5a3050c702b552eb045c02d0cf9cb83c7/src/Hooks.php#L166 | their server-side implementation ]] with me. In summary: > - The code makes use of a helper function [[ https://github.com/wikimedia/mediawiki-extensions-ReaderExperiments/blob/043716c5a3050c702b552eb045c02d0cf9cb83c7/src/Hooks.php#L66 | isInAnyTreatmentGroup ]] which checks if the request is assigned to any group for a given experiment > - If the request is assigned to any group and satisfies other conditions (e.g. main namespace, skin), it loads `ext.readerExperiments.imageBrowsing` module and also adds `<div id="ext-readerExperiments-imageBrowsing"></div>` to the response. > - Therefore, all users enrolled in the experiment would have this client-side module loaded. > - The module then checks for enrollment/assignment and > - If user is in control group, do nothing to `div#ext-readerExperiments-imageBrowsing` > - If user is in treatment group, populate `div#ext-readerExperiments-imageBrowsing` > With this pattern, they could easily call `Experiment#logExposure()` on the client-side and have it be accurate (only log exposure when there is actually exposure to any variation). > > So they effectively have a solution to this task simply by only loading & running client-side code using server-side logic. As long as they continue to use this pattern for implementing their experiments, they're all good. We can provide a recommendation for how to log exposure on the client-side, namely by using server-side logic to conditionally load client-side instrumentation which would include an exposure logging call. ## Acceptance criteria [ ] https://wikitech.wikimedia.org/wiki/Test_Kitchen/Conduct_an_experiment includes: - examples/scenarios - do's & don't's - recommended patterns / best practices