User Story
As the Growth team, I want to define and measure key metrics that assess whether the redesigned logged-out editing "warning" changes user behavior, increases account creation, and impacts abandonment, so that the Growth team can determine if the intervention should be rolled out more broadly.
Measurement Goals
Evaluate whether softening the logged-out warning message and reframing the call-to-action (CTA) leads to:
- An increase in account creation from users who initiate an edit while logged out.
- A decrease in editing abandonment after encountering the logged-out warning.
- A reduction in the proportion of edits made under temporary accounts compared to full account creation.
- Improvements in user engagement and retention for those who create accounts via this flow.
Acceptance Criteria
Metrics to track:
| Metric | Description |
|---|---|
| Experiment exposures | Number of impressions of the relevant interface for users enrolled in the experiment |
| Create Account link clickthrough | For each user, proportion of users who saw the Create Account link and the impression resulted on a click. |
| Log in link clickthrough | For each user, proportion of users who saw the Log in link and the impression resulted on a click. |
| Edit without logging in clickthrough | For each user, proportion of users who saw the Edit without logging in link and the impression resulted on a click. |
| Temporary Account Learn More link clickthrough | For each user, proportion of users who saw the Temporary Account Learn More link and the impression resulted on a click. |
| VE close button clickthrough | For each user, proportion of users who saw the VE close button link and the impression resulted on a click. |
| Constructive edit rate (mobile web) | Proportion of main namespace edits made on a mobile browser that are constructive (not reverted within 48 hours of being published). |
| Constructive edit rate of newer editors (mobile web) | Proportion of main namespace edits made by a mobile browser-using editor with 100 or fewer edits that are constructive (not reverted within 48 hours of being published). Editors who had 100 or fewer edits going into the experiment but exceeded the threshold by the end of the experiment (by making more edits) are not excluded -- that is, they do not "graduate" out of the experiment cohort. |
| Account creation rate (primary metric) | For each user, proportion of users who were exposed to the experiment entry point, and the impression resulted on a successful registered account. |