In order to associate user interactions (event data) to experiments, we need a way of recording that membership data. We also need that way to support membership in multiple concurrent experiments. After evaluating two potential data modeling approaches based on complexity of analysis and implementation, ability to extend the model in the future, and performance considerations, we decided that membership should be recorded using the following data model:
experiments: struct< enrolled: array<string>, assigned: map<string, string> [, [property]: map<string, [type]>] >
where:
- enrolled is an array of names of all experiments that the subject is currently enrolled in
- assigned is a map of which experiment groups (variants) the subject has been assigned to
- the key is the experiment name present in the enrolled array
- the value is the assigned experiment group (variant)
- any additional properties would store additional data about the experiments using the same model as assigned, where the keys match the values found in the enrolled array.
More details in the decision record.
Technical details
Schema updates
A new experiment fragment must be created ( (according to the instructions above) ) to be added to the common one. After that, the new common fragment will be used to create a new version for base schemas (app and web). Integration with common and base fragments will be done at T366802: Update Metrics Platform Base Schemas to include instrument name
JS Client library
- We need to create a new function to get the current user experiments membership to be able to add that information automatically to every produced event. The function must be created there where Metrics Platform is integrated, so we need to define it in the Integration interface and provide an implementation where needed. A default one for DefaultIntegration.js and specific ones for MediaWikiMetricsClientIntegration in the case of the JS client library. The new function will be used as a part of the submitInteracion() function to be able to add that information to every event row.
- We also need to create another function that provides the assigned bucket value for a specific experiment for the current user. This function will be used by instrument developers to know which variant has been assigned to the current user and, that way, they will know what needs to be implemented for every case. Something like getExperimentBucketValue(experimentName) that returns the bucket value that is assigned for the current user. Also as a part of the integration with Metrics Platform, we should add this function to the same place as before
PHP Client library
TBD
Requirements
- Schema updates
- Schema fragment (merged)
- Integrated into base schemas (T366802: Update Metrics Platform Base Schemas to include instrument name)
- Client library updates
- Documentation updates
Writing MediaWiki instrumentation code using the EventLogging extension(to be deprecated as part of docs refactoring)Writing MediaWiki instrumentation code for apps(deprecated as part of docs refactoring)- https://wikitech.wikimedia.org/wiki/Metrics_Platform/JavaScript_API