Motivation
We want to analyze the data captured by the instrumentation in T418336: Banner to encourage user to confirm their email and T415902: Instrument data for tracking email verifications.
Specifically, we want to calculate:
- Click-through rate (number of clicks / number of impressions)
- Daily frequency
- Broken down by wiki
- The impact of the banner on increase in email confirmations:
- Daily frequency
- Broken down by wiki
Hypothesis
Showing the email confirmation banner will increase the rate of email confirmations compared to not showing the banner.
Instrumentation
The following events will be collected via Test Kitchen:
- impression (banner is shown)
- click (user clicks confirmation link)
These events will be used to compute click-through rate and correlate banner exposure with existing email confirmation events.
Analysis approach
- Compute click-through rate over time
- Measure change in email confirmation rates over time and in relation to banner exposure
- Compare trends to identify differences in impact
Data collection risk tier: Low
Instrumentation specification
- Schema: analytics/product_metrics/web/base
- Stream: product_metrics.web_base
- Events:
- impression (banner is shown)
- click (user clicks confirmation link)
- Contextual attributes:
- Handled automatically by Test Kitchen
- Sampling:
- Initial rollout at % of sessions (TBD)