Background
To effectively measure the impact of adding more dynamic improvements to the tabbed browsing feature, we would like to run it as an A/B/C test with a limited audience.
Requirements
- Wrap up the new tab experience
- Sort the users into as soon as possible after opening the app (do not require closing of the app for sorting)
- Limit the experiment audience to:
- Users with an App primary language of English, Arabic, or German
- Users with a device country of Germany, or within ESEAP or MENA regions (use these lists)
- Stop sorting new users into experiment on 15 January. All other users should retain their experiences until we make a scaling decision
- Test set up
- A group
- No changes to tab behavior, users still must have 1 tab open
- New tab remains the Main Page, search is not active on opening a new tab
- B Group
- On first launch of tabs users should see a Main Page tab in their overview (which can now be closed).
- If a user opens a new tab it should take them into a focused search
- C Group
- There is no "Default" open tab of the main page. Users start with an empty state (0 tabs)
- When opening a new tab it brings them to main page, search is not active
- BOTH Group B & C
- Users can have 0 tabs. If someone closes all tabs, they see an empty state
- Add a module at the bottom of the tabs screen that has Did You Know if they have less than 2 tabs and showing Because You Read content based on their most recent tab should they have 2 or more tabs open.
- Empty state illustration and text
- Recommended content pinned to the bottom of the tabs overview screen
- Overflow menu with the ability to hide recommendations AND close all tabs
- Highlight the active tab
- Show the survey
- A group
QA Notes
The most "pure" test of these would be:
- Set your device region into Germany, or within ESEAP or MENA regions
- Fresh install
- Set the target App primary language to English, Arabic, or German in onboarding.
- When you land on Explore, tap the tabs overview button.
- If you see an initial Main tab, then you are in either Group A if there is no close button, or Group B if there is a close button. If you see an empty state, then you are in Group C.
We have developer settings toggles for each experience, but they are a little awkward when it comes to testing the initial tab overview state. This is because the app tries to immediately create a Main Page tab in Groups A and B. If you then switch the toggle to Group C via developer settings, the tabs overview initial state will still show the Main Page tab, because we aren't deleting any tabs when we switch the developer settings toggles.