Summary:
We are running an experiment to improve the mobile account creation form on Special:CreateAccount (auth.wikimedia.org). However, the current Test Kitchen architecture prevents us from collecting instrumentation data for an experiment from both auth.wikimedia.org (where the user actually encounters the Special:CreateAccount form), and the origin wiki (where we would be able to record whether an account was actually created, and record metrics like constructive edit rate).
Background:
Hypothesis:
If we improve the account creation form, the registration completion rate among mobile users who land on the form will increase by at least 2% relative to the control group.
Current limitation:
- The experiment runs on auth.wikimedia.org (Special:CreateAccount)
- Account creation success is only recorded on the origin wiki
- There is currently no reliable way to connect:
- exposure to experiment variant (auth domain)
- successful account creation & down-funnel contributor metrics (origin wiki)
Proposed approaches (to evaluate and implement):
Option A: Cross-domain experiment attribution
- Pass experiment assignment (e.g., bucket + experiment name) from auth domain to the origin wiki during account creation flow
- Persist this information (e.g., via request parameters, session, or temporary storage)
- Log successful account creation events with experiment attribution
Option B: Server-side logging on account creation
- Emit a structured event when an account is successfully created
- Include:
- experiment name
- variant (control/treatment)
- originating wiki (e.g., enwiki)
- Requires propagating experiment assignment to the point of account creation
Option C: Metrics pipeline workaround (Prometheus/Grafana)
- Track aggregate account creation counts correlated with experiment rollout
Example implementation pattern:
To target auth domain in TestKitchen UI and then check wgDBname for "enwiki" (for example) before using TK SDK, e.g.
PHP (server-side):
if ( $someTargetingCondition ) {
// ^ needs to be consistent with the targeting check done client-side
$experiment = ...
if ( $experiment->isAssignedGroup( 'treatment' ) {
// code for varying the form
}
}JS (client-side):
if ( someTargetingCondition ) {
// ^ needs to be consistent with the targeting check done server-side
const experiment = mw.testKitchen.getExperiment( EXPERIMENT_NAME );
// user would have been shown control or treatment variation on this page load, as determined by server-side check
experiment.sendExposure();
}Acceptance Criteria:
- We can measure total registrations or registration completion rate for control vs treatment
- Experiment is restarted and can scale to a larger percentage of mobile traffic