Background

To effectively measure the impact of adding Tabs, we would like to run it as an A/B test with a limited audience.

Requirements

• Limit the display of Tabs to these audiences who meet these criteria
  • Users with an App primary language of English, Arabic, or Japanese
  • Users with a device country within ESEAP or MENA regions (use these lists)
• Within that audience, only 50% of users should see the entry point for Tabs
• Sort users into the test upon first article view (not app launch, ensure testers can reach developer settings)
• Stop sorting users into control and treatment on July 31
• Tabs is not removed at end of test
• Confirm with Instrumentation Deck and @SNowick_WMF on labeling of groups

Engineering Notes

• Create A/B test logic (group assignment, post assignment to analytics). Check A/B test logic before displaying tabs entry point icon and long press options.

Testing notes

(A/B tests assignment testing)

Change the device region to a qualified region.

• Fresh install app. Choose a qualified primary app language in onboarding
• Tap into an article, you may or may not see the tabs icon.
• Repeat steps 2-3. Ensure half of the time you see the tabs entry point, the other half you don't.

(Dev settings testing)

• Fresh install app, this time on an unqualified device region or primary app language.
• Ensure you don't see the tabs entry point.
• Go to developer settings and force on the tabs toggle.
• Ensure you do see tabs entry point.

(Date testing)

• Temporarily change the end date to a date near in the future.
• Fresh install on a qualified device region and primary app language. Repeat until you see tabs entry point to confirm you are in the test assignment.
• Change device region to after end date in step 9.
• Re-enter the app, you should still see the test behavior (tabs entry point).
• Fresh install the app, relaunch. Ensure you cannot be assigned into test bucket.