Scope
- What would generalized enrolment authority look like?
- What are its enrolment authority in relation to a different enrolment authority?
- What would an API look like for enrolment on a custom token?
- What are the responsibilities and what should it do?
- Is non cache splitting just another enrolment authority?
Obvious Gotchas
- If Test Kitchen enrols people automatically on global identifier, what does it mean to enrol based on local identifier? How do we represent that?
- Don't be influenced by what's currently written - what is the ideal state and how do we move backwards from that?
- Conceptual doozie
Outcome
What do they do?
- An experiment enrollment sampling authority ensures that all group assignments are equally likely for all experiments
- For example, if there are two experiments running, both with control and treatment groups, then the user should be enrolled into the experiments in such a way that the following experimental group assignments are equally likely:
| Experiment 1 | Experiment 2 |
|---|---|
| control | control |
| control | treatment |
| treatment | control |
| treatment | treatment |
- An EESA consistently assigns the same user to the same experimental group for the same experiment
What don't they do?
- Manage the identifiers that represent the users
- Coordinate the enrollment of a user into multiple experiments, which may or may not require different identifiers
- Decorate the output with metadata about the enrollment of the user into experiments
(1) may or may not be performed by an external component. (2) and (3) will be performed by the Coordinator and Decorator components, respectively.
Note well that (2) and (3) will be performed automatically to enable CSS-only experiments. CSS-only experiments are those that only vary the style of a feature based on CSS classes added to the <body> tag by Test Kitchen.
Example 1: T405074 xLab: Allow user re-enrollment at specific times
Test Kitchen enrolls the user in experiments that require the mw-user identifier type in the BeforeInitialize hook using the central user ID. Unfortunately, the Growth team discovered that the central user ID isn't available to BeforeInitialize hook handlers during the account creation flow. In order to work around this limitation, the Growth team reimplemented the enrollment coordination and output decoration steps. They had to do this because Test Kitchen doesn't separate EESAs from the enrollment sampling coordination and output decoration steps.
Recommendation #1: Defer Output Decoration
Currently, the output decoration step is performed immediately after the experiment enrollment sampling step in the BeforeInitialize hook handler. It could and should be performed much later in the BeforePageDisplay hook handler. This is consistent with (3) in § What don't they do? above.
Recommendation #2: Allow Identifiers to be Updated
In order to handle the cases where an identifier type isn't available or has been invalidated, the experiment enrollment coordinator should allow identifiers to be updated. Unfortunately, hook handlers are called in the order in which they are registered so we should provide an API.
PHP
namespace MediaWiki\Extension\MetricsPlatform\TestKitchen; interface Coordinator { const IDENTIFIER_TYPE_EDGE_UNIQUE = 'edge-unique'; const IDENTIFIER_TYPE_MW_USER = 'mw-user'; const IDENTIFIER_TYPE_SEARCH_SESSION = 'search-session'; /** * Updates the identifier. * * When an identifier is updated all experiments that require the * identifier type are enrolled and the `XLab.ExperimentManager` service is * updated with those enrollments. If this is done before the * `BeforePageOutput` hook is run, then the output is decorated with those * enrollments also. * * @throws \DomainException If the identifier type can't be updated, i.e. * it's {@link Coordinator::IDENTIFIER_TYPE_MW_USER} */ public function updateIdentifier( string $identifierType, string $identifier ); }
JavaScript
enum IdentifierType { EDGE_UNIQUE = 'edge-unique', MW_USER = 'mw-user', SEARCH_SESSION = 'search-session' } interface Coordinator { /** * Updates the identifier. * * When an identifier is updated all experiments that require the * identifier type are enrolled and the body is decorated with CSS classes * corresponding to those enrollments. * * @throws Error If the identifier type can't be updated, i.e. * it's {@link IdentifierType.EDGE_UNIQUE} or * {@link IDENTIFIER_TYPE.MW_USER} */ updateIdentifier( identifierType: IdentifierType, identifier: string ): void; }
Recommendation #3: Generalize Output Decoration
It's conceivable that Test Kitchen could decorate the output of the MediaWiki Action and REST APIs. Test Kitchen should select and use the appropriate output decorator.
Quoting T405074:
Another use cases for Growth that has come up is the retrieval of an assigned group within an action API or rest request.
Use case(s) (list the steps that you performed to discover that problem, and describe the actual underlying problem which you want to solve. Do not describe only a solution):
- Experiment assigned group retrieval on account creation
- Experiment assigned group retrieval on API calls
- Experiment assigned group retrieval on maintenance scripts
Uses cases (1) and (2) are covered by recommendations (1), (2), and (3) above. However, those recommendations assume that the MediaWiki application is responding to an HTTP request, which doesn't hold in use case (3). Maintenance scripts don't have BeforeInitialize and LocalUserCreated hook handlers, they don't have output to decorate, and, crucially, they are userless.
Recommendation #4: Expose Experiment Enrollment Sampling
namespace MediaWiki\Extension\MetricsPlatform\TestKitchen; interface Coordinator { /** * Gets the experimental group assignment for the identifier. * * This method is pure – it has no side effects. * * @return {string|null} If the user is enrolled in the experiment, then * then the name of the experimental group; otherwise, `null` */ public function getEnrollmentForIdentifier( string $experimentName, string $identifier ): ?string; }
Example 2: Experiments on Search Sessions
In § Example 1 above, we recommended upgrading the experiment enrollment coordinator in Test Kitchen to handle cases the cases where an identifier type isn't available or has been invalidated. Another motivating example for this upgrade is experiments on search sessions via Test Kitchen.
Currently, there are two distinct classes of experiment that can be run on search sessions: (1) experiments on the user interface; and (2) experiments on the backend used by the CirrusSearch MediaWiki extension ("CirrusSearch"). (1) has been handled on a per-experiment basis whereas (2) has been handled by the experiment enrollment coordinator in CirrusSearch. If we follow Recommendation 2 in § Example 1 above, then (1) should be trivial to implement:
// ext.testKitchen/index.js // // Maintained by the Experiment Platform team function onNewSearchSession( searchSessionIdentifier: string ) { mw.testKitchen.updateIdentfier( mw.TestKitchen.IdentifierType.SEARCH_SESSION, searchSessionIdentifier ); } // --- // ext.wikimediaEvents/experiments/search_experiment_1.js // // Not maintained by the Experiment Platform team function onNewSearchSession() { mw.testKitchen.getExperiment( 'my-awesome-search-experiment' ) .send( 'exposure', { action_context: e.getAssignedGroup() } ); }
However, (2) is more complicated. There are four scenarios that the experiment enrollment coordinator in CirrusSearch handles. The first scenario, "[s]ession starting on-wiki with autocomplete", which is the most common, could also be handled by the example code above, but the remaining three need careful consideration.
Conclusion
Ostensibly, this spike is exploring a generalized EESA. However, we have explored a handful of real-world examples and found that:
- EESAs are already sufficiently general
- Test Kitchen has a hidden object, the Experiment Enrollment Coordinator
- Experiment implementors have been using the Experiment Enrollment Coordinator
- It's possible and desirable to expose the EESA to experiment implementors without comprimising the functionality of the Experiment Enrollment Coordinator