Page MenuHomePhabricator

[S] Activity tab (user story): Experiment setup
Open, MediumPublic

Description

User story

As a product manager, I want to run this feature as an A/B test with a survey and instrumentation so that I can communicate the results and impact of this feature. =

Requirements
  • Wrap up the new activity tab in an A/B test
  • Randomly sort all users into groups
    • 50% into A: History tab (no change)
    • 50% into B: Activity tab V2
  • On January 15
    • Stop sorting new users into groups
    • Users with group B should keep the activity tab v2 (because we plan to scale)
  • Audience: all users
  • Instrumentation plan created to answer the KRs within the Epic
  • Feature is instrumented according to deck (may be broken up by release if needed)
Reference

Deep dive: https://docs.google.com/presentation/d/1SRpZc9NaShgb6XndhwrC966n3hTSHKntZGfpI3dSdQc/edit?slide=id.g35f04951676_0_0#slide=id.g35f04951676_0_0

Engineering notes

We should have 3 dev settings toggles to carefully control things. One is our existing feature flag toggle, then two new toggles for forcing a particular group assignment.

  1. Activity tab feature flag dev toggle > this enables the app to run any activity-tab-related code.
  2. 2 new dev toggles for forcing an assignment here, control & activity tab v2.
  3. When the app launches: A. If feature flag is enabled, we go through group assignment logic i. If control is forced here via dev settings toggle, assign control bucket ii. if experiment is forced here in dev settings toggle, assign experiment bucket iii. If neither are forced, go through the randomized 50/50 assignment bucket B. When setting up the root view, we check the feature flag. If not enabled, embed History. If enabled, check the group assignment, if control embed History, otherwise embed Activity tab

This means, once this task is done, to get to Activity tab we have to:

  1. Fresh install, go to dev settings, enable feature flag toggle, then enable forcing experiment assignment toggle.
  2. Terminate + relaunch the app, activity tab should show.

Event Timeline

HNordeenWMF renamed this task from Create Activity tab A/B/C test to Create Activity tab V2 A/B/C test.
HNordeenWMF updated the task description. (Show Details)
HNordeenWMF renamed this task from Create Activity tab V2 A/B/C test to Create Activity tab V2 A/B test.Fri, Oct 24, 1:09 AM
HNordeenWMF updated the task description. (Show Details)
HNordeenWMF updated the task description. (Show Details)
HNordeenWMF renamed this task from Create Activity tab V2 A/B test to Activity tab (user story): Experiment.Tue, Oct 28, 10:28 PM
HNordeenWMF renamed this task from Activity tab (user story): Experiment to Activity tab (user story): Experiment setup.
HNordeenWMF updated the task description. (Show Details)
Tsevener renamed this task from Activity tab (user story): Experiment setup to [S] Activity tab (user story): Experiment setup.Thu, Oct 30, 1:56 PM

@HNordeenWMF

Randomly sort all users into groups
A: History tab (no change)
B: Activity tab V2

Do you want to bucket 50% in each I assume? We could skew it more towards one group too if you'd like.

Also, just a heads up, the root view setup for activity tab happens immediately on launch, so the group assignment also should happen upon launch (before we assign the root view).

Yep 50/50! @Tsevener sorting on launch works, assuming we'll have a dev setting to change groups for QA?