Background

• Release date: Dec 14
• 15 Days - Dec 29

The Task

• Compare results to baseline data that was collected
• Visualize and present the data in a way that is easily understandable to the team and answers the hypothesis

Hypothesis

If we A/B test adding an “Activity” tab on iOS that highlights reading, editing and other participation behaviors, we will see a 1% increase in overall app retention compared to our control experience of “History”.

At 15 days:

Answer the key results first for the hypothesis:

• Primary KR 1.1 - Users with access to the activity tab have a 1% higher overall app retention rate among logged-in readers of experiment group compared to control
• KR 1.2 - 5% higher multi-day visits of activity tab among logged-in users who have the new activity tab vs those that have the history

Check guardrails

• GR 2.1 No more than 10% of users turns feature completely off
• GR 2.2 At least 80% of qualitative feedback received is positive or neutral

Check curiosities (nice to have)

• CR 3.1 – What % of people switch their Activity Tab experience to something other than the default? Which modules are people turning off and at what rate?
• CR 3.2 - By how much did account creation increase for the experimental group compared to control? What did the log-in/account creation funnel look like?
• CR 3.4 - How many new accounts took an action on the activity tab? View an article, or making an edit
• CR 3.5 - How does engagement with “History” compare with engagement with Timeline on Activity?
• CR 3.6 - How does the return rate compare to the Activity tab V1 prototype’s return rate?