Motivation
We want to A/B test the banner presence before a complete rollout to ensure there is no unintended negative impact on the user behavior.
Hypothesis
- We see an increase in confirmed emails for users who see the banner
Experiment setup
- Users: Logged-in users with unconfirmed emails
- Wikis: English, French, German, Polish, Japanese, Hebrew, Wikidata, Commons, Mediawiki, Meta
- Experiment setup:
- A/B test
- Arm A: 50% of users in the experiment see banner version 1
- Arm B: 50% users in control group
- Length of experiment: 2 weeks (summary of power analysis)
Metrics
- Primary metric: Number of accounts with confirmed emails
- Secondary metric: Banner impressions and Banner clicks
- Guardrail metric: Number of accounts with emails
Results
- We observed increase in email confirmations across all projects. On average we saw a 92.6% increase in email confirmations on the projects where we had substantial data.
- Email removal ranged from 0.15% to 1.9% on the projects. These numbers are low so we don't have a significant statistic to report but it was higher than the control group. Worth noting that about 25% of users who removed their emails went on to confirm them later indicating that they were replacing their old email with a new one.
On the whole, this experiment is considered a success and we will proceed with rolling out the banner to all eligible users on all projects.
