User stories
As someone who read the Katherine Goble article and the NASA article, I want to receive the option to generate a list of recommended of articles to read related to Katherine Goble and NASA, so that I can save the list and read the articles later.
As someone that uses the Android app on a regular basis, I would like my recommendations to be refreshed on a weekly basis based on my updated reading history, so that I can regularly discover new articles.
Key Indicators
Measure after 7 & 30 days:
- KR1.1 Unique logged-out Treatment Group users open the app on multiple days within a 20-day period at a rate at least 1% higher than those in Control Group
- KR 1.2 5% increase in Reading list engaged retention measured for the experiment group relative to control engaged
- KR 1.3 5% increase in avg daily internal referral clicks measured for the experiment group engaged relative to engaged control
- CUR1: 5% increase in share of users interacting with reading lists measured for the engaged experiment group relative to engaged control
Definition of Control reading list users
- Interactions with reading lists defined as a user which saves two or more articles or opens the saved screen twice or more
Definition of Variant reading list users
- Interactions with reading lists defined as a user which saves two or more articles or opens the saved screen twice or more
- OR
- Opened the discover list twice or more
All should have splits for logged-out vs logged-in.
Measure after 30 days:
Curiosities (Only for 30 day results) From Measurement Plan
- 5% increase in share of users interacting with reading lists measured for the experiment group relative to control
- Do we see a difference in the retention rate for logged in users vs logged out users?
- What % of users that engaged with the feature take an action from the recommended list broken down by: viewing, sharing, or saving to a reading list?
- What percent of users select No thanks in Recommended Reading List prompt?
- How many articles do people select for their interests (distribution of counts) ? Only in selection active_interface NOT in Settings (First time is only time unless user goes to settings)
- What % of “Discover” views came from users opening notifications?
- How many people are enabling or disabling notifications for reading lists when prompted [are users being prompted here?], or in settings? What percent uniques change setting?
Added Questions TBD:
(Only for 30 day results) Not from Measurement Plan
- 1% increase in app-retention measured for the experiment group relative to control
- What are people's preferred choice for source of recommendations?
- Notifications: what % of “Discover” views came from users opening notifications?
- How many people are enabling or disabling notifications for reading lists when prompted, or in settings?
- How many people disabled the “discover reading list” when first prompted or in Settings?
- How many articles do people select for their interests?
- What % of users that engaged with the feature take an action on an article from the recommended list broken down by: viewing, sharing, or saving to a reading list?
- Does different recommendation generation have differing effects on retention?
Must haves
- Ability for user to decide which article they’d like recommendations from
- Ability for user to request that we generate a list for them
- In cases where we choose for them, we should use their most recent article in their history
- Info page to learn more about the feature
- Ability for users to save their reading list and let them know they can add and delete articles from their list once saved
- Clearly inform users that their list will be updated weekly by their most recent reading history but they can modify by selecting article of interest
- Allow users to turn off/hide recommended reading lists
- Use indicator above Saved tab so users know that they have recommended reading lists available. Only users that have reading lists but have not clicked a suggested reading list should see the indicator. Once user click the suggested list the indicator should go away
Nice to haves
- Allow users to generate an article using more than one article
- Allow users to turn on reminders that their new list is ready
- Update cadence of suggested articles
- Modify day that it would go out.
Timeline 🚀
- Apr: Finalize Designs
- May: Development
- Beginning of June: Release Everywhere
- Mid-June: Evaluation (at least 15 days worth of data)
Design
Core Screens
