Background
This task is intended to capture what data wiring is needed to answer if this feature met its goals.
Measurement Goals [This will be updated pending discussion of revised Measurement Plan]
Validation
These will be measured via an A/B test
- KR 1.1 8% increase in users initiating Search from article view
- Control group: unique users in A who have initiated at least one search from article view/unique users in A who have had article views
- Variant group: unique users in B who have initiated at least one search from article view/unique users in B who have had article views
- [Primary metric] KR 1.2 1% higher app retention rate for logged-out users that had the more prominent article search (using 2-week retention)
- Control cohort: of the unique users in A who engaged with the control search icon, what % were retained to the app?
- Variant cohort: of the unique users in B who engaged with the variant search bar, what % were retained to the app?
- KR 1.3 5% increase in average pageviews/unique user for those who engaged new article search bar
- Control group: average pageviews/unique user for users in group A who initiated at least one search from article view
- Variant group: average pageviews/unique user for users in group B who initiated at least one search from article view
Guardrails
- GR 2.1 No more than 10 negative user complaints about changes to the Navigation that can’t be addressed
- GR 2.2 No more than 5% increase in app uninstalls (vs baseline)
- GR 2.3 No more than a 10% increase in app crashes
Curiosities
- CR 3.1 Did account creations increase as a result of the Profile menu being accessible from all screens? (Baseline of 15 days before vs Experiment Date Range)
- CR 3.2 Do users prefer a different location for the article view search?
- CR 3.3 What was the feature retention rate for article search for both A & B? (Recalculate 1.2 but using return to engage with article search as definition of retention, not return to overall app) Please split by Logged-in and logged-out
Link to Finalized Designs
- Navigation Part 2 designs: https://www.figma.com/design/utNpL8yrjrDkDmlO9qS4T5/iOS-Navigation-(Robin)?node-id=679-745&p=f&t=YUiSsRUkT16A9s2k-0
- Search prominence A/B test will look like this:
- Tasks
- @ PM Add measurement goals to task
- @ PM Add link to final designs
- @Analyst Create schema Doc
- @analyst Create validation task after Beta release