#### Background
As a part of our annual plan we are focused on increasing the retention of a new generation of readers on our website, allowing a new generation to build a lasting connection with Wikipedia, by exploring opportunities for readers to more easily discover and learn from content they are interested in.
We are beginning the fiscal year by experimenting with a series of experiments of browsing experiences to determine which we would like to scale for production use, and on which platform (web, apps, or both). We will then focus on scaling these experiments and testing their efficacy in increasing retention in production environments. Our goal by the end of the year is to launch at least two experiences on representative wikis and to accurately measure a 5% increase in reader retention for readers engaged in these experiences.
#### OKR Hypothesis
This work is apart of the [[https://meta.wikimedia.org/wiki/Wikimedia_Foundation_Annual_Plan/2024-2025/Product_%26_Technology_OKRs#Draft_Key_Results | 2024-2025 Annual Plan Wiki Experiences 3.1]] work.
**Official Hypothesis: **
If we introduce a personalized rabbit hole feature in the Android app and recommend articles based on the types of topics and sections a user is interested in, we will learn if the feature is sticky enough to result in multi-day usage by 10% of users exposed to the experiment over a 20-day period, and a higher pageview rate than users do not engage with the feature.
**Secondary Hypothesis:**
If we compare rabbit holes that are accessible based on initiation in the search field vs. through readinglists we will see higher engagement rates and retention with content presented in the search as opposed to reading lists.
#### How will we know we were successful
**Validation**
10% of unique users that engage with experiment do so more than once in a 30 day period
5% higher internal referral clicks of recommended articles from people that engage with the experiment compared to those that did not engage with feature
5% higher clicks on suggestion based on search queries vs. click throughs to articles from suggested reading lists
5% higher clicks to view suggested reading list vs users that hit enter on search queries
65% or higher of feedback scores are positive
**Guardrails**
60% of feedback from in-app users are negative
10 community members from target group express negative sentiments
**Curiosities**
Do we see a difference in the retention rate for logged in users vs logged out users
How does the metrics from Rabbit hole compare to Recommended Content
Do we see higher pageviews for users that engage with the feature vs those that do not
####User Stories
**Reading List**
As someone who read the Katherine Goble article and the NASA article, I want to receive a list of recommended of articles to read related to Women in Science and NASA, so that I can come back and read them later.
**Search**
As someone who is reading the Philanthropy section of the Beyonce article, I want to see a recommendation prepopulated in search that reads “Beygood Partnerships” and when I press search it provides highlights of the organizations BeyGood has partnered with and I can decide to read more about those partners or go to another article, so that I can decide which article I want to go deeper into.
**In-App Feedback**
As a tester of the rabbit hole experiments, I would like to provide feedback about recommendations without leaving the app, so that the feedback gets to the dev team and recommendations can improve over time.
####Must Haves
- Run as ABC test
- A group is the control
- B group sees a recommended search query from article view in the search
- C group receives dialog encouraging them to see and save their recommended reading list
- Interface should be clear that recommendations are based on user interest
- Constrained to Sub Saharan Africa and South Asia
- User able to provide feedback about the quality of recommendations in-app
- Recommendations should pull from Categories, Topics or MoreLike. When doing instrumentation, there must be a way for us to know which selection a user made was from which API.
- After 20 days the experiment should be removed from the app
####Nice to Haves
- Display summaries of articles in search and reading list interface
#### Target Quant Regions and Languages
South Asia & Sub Saharan Africa
**User Testing Languages**
- English
- Hindi
- French
- Arabic
**User Testing Considerations**
- Impact for screenreaders
- Impact for RtL readers
- Preferences based on Age
####[[ https://www.figma.com/design/Z2n0gZOOAbgsIdHgUS5ocQ/Android-%E2%86%92-Rabbit-Holes-%E2%86%92-T378612?node-id=0-1&t=3VNcjKuDI59BVi5r-1 | Designs ]]
**Reading list**
| **Announcement dialog** | **Reading list** | **Saved screen (list=unsaved)** | **Saved screen (list=saved)** | **Feedback dialog**
| {F57717583} | {F57717585} | {F57717588} | {F57717591} | {F57717594}
**Rabbit holes**
| **Suggested search query in-article** | **Search screen (initial)** | **Search screen (active)** | **Feedback dialog**
| {F57717604} | {F57717606} | {F57717608} | {F57717611}
#####Qual Research Questions by Designs
**Reading List**
- Should reading lists be dynamic?
- If so what cadence should they update
- Do people want reminders that their reading lists have been updated?
- Do people want us to guess their reading list based on history? How does that weigh against them setting categories/topics they are interested in and getting list based on those topics?
- If they want to select what they are interested in, how should they adjust their interests?
- Should there be articles recommended to be added to their existing lists based on its current list?
**Rabbit Holes**
- Should the query recommendation take users straight to an article or launch search?
- How do people feel about seeing a summary with the query?
- Should the cursor be at the front or the back of the query?
- Should we combine elements of rabbit holes and recommended content?
####Resources
Work in Progress Product [[ https://docs.google.com/presentation/d/1HMhL5KkksogHOSdMGfYB46wGInsveW8iUX_-teXltYM/edit#slide=id.g23ce2a303f9_0_0 | Deck ]]