B Test
Closed, ResolvedPublic3 Estimated Story Points
Actions

Assigned To

Authored By

	HNordeenWMF
	May 29 2026, 8:39 PM

Project Tags

Referenced Files

None

Subscribers

Description

Background

We're redesigning the Explore feed: a two-tab experience where 'Community' surfaces Wikipedia's best editorial content in a familiar, scannable format, and For You delivers a full-screen, immersive experience personalized to your interests, reading history, and location.

We need to be able to measure the impact of this improvement on retention and engagement.

Requirements

The release with the Explore Feed revamp should only be made availalbe to 50% of iOS app users
- A: Control (Explore tab, no changes to feed)
  - Existing onboarding
- B: Variant with redesigned feed
  - Renamed to Home
  - Updated onboarding with Interest selection for new installs and existing app users
  - “For you” feed with new language selector
  - Additional settings for "For you"
  - Existing Explore Feed wrapped into “Community”, With any modules that have been added to "For you" removed
Audience: All languages, logged-in and logged-out
Analytics must be able to measure retention to the app for Group A and Group B
Test should have no end-date. We will create a scaling task to end the test.

Note: follow-up releases will contain

Additional “For you” modules (Random, Places, Games), removing from legacy feed as we go
Rewritten “Community” tab

Test Notes

Once this work is in, re-test https://phabricator.wikimedia.org/T427577 as well. Complete test notes:

Fresh install latest TestFlight RC. IF:

User is assigned control, you will see old legacy app onboarding. Upon completion of onboarding, you should NOT see one-time onboarding in T427577.
User is assigned test, you will see new app onboarding. Upon completion of onboarding, you should NOT see one-time onboarding in T427577.

Delete the app, fresh install current App Store App, go through old legacy app onboarding. Then, upgrade to the latest RC via TestFlight. Launch. IF:

User is assigned control, you will NOT see one-time onboarding in T427577.
User is assigned test, you WILL see one-time onboarding in T427577

Toni Handoff Notes

Based on progress at PR https://github.com/wikimedia/wikipedia-ios/pull/5995

This work has not been done yet.
We won't be adding test kitchen integration.
Just a note that we need to add A/B testing ability into WMFTestKitchen, so this requires some plumbing work before it becomes simple. To build out the original iOS WMFTestKitchen, we asked Claude to reference Android's and port it over to iOS. We can take the same approach to add the experiment pieces to WMFTestKitchen.

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Open		None	T427526 [Epic] Scale Home Feed to iOS (Explore Feed Revamp)
		Resolved		GOlson-WMF	T427688 [M/L] Deploy Onboarding & "For you" feed revamp as A/B Test

Event Timeline

@Engineers, and @Seddon please discuss the feasibility of both options here: traditional A/B test vs holdout release

HNordeenWMF moved this task from Backlog to P5 (5 sprints out) on the Explore Feed Refresh iOS board.May 29 2026, 8:50 PM

HNordeenWMF moved this task from P5 (5 sprints out) to P4 (4 sprints out) on the Explore Feed Refresh iOS board.Jun 16 2026, 5:40 PM

Tsevener updated the task description. (Show Details)Jun 26 2026, 10:56 PM

HNordeenWMF edited projects, added: App Experience; removed: Wikipedia-iOS-App (Wikipedia iOS Release FY2025-26).Jun 29 2026, 1:34 PM

HNordeenWMF moved this task from Team Triage to Backlog on the App Experience board.Jun 29 2026, 2:13 PM

@HNordeenWMF since we won't be using distinct app versions for this AB test we will not be able to identify Uninstalls by App version to differentiate Uninstallers - we can do a pre/post analysis to look at if Uninstalls increased after feature release.

M/L - pending deeper review of TestKitchen AB testing capability integration

@Mazevedo @Seddon do ya'll need to create a spike for that?

HNordeenWMF updated the task description. (Show Details)Jul 2 2026, 8:45 PM

Hi All - I met with Mikhail today and we will not be able to integrate this feature with GrowthBook/TestKitchen for various reasons, will link docs and explanation when I get it from Mikhail. We will just use the experiments."assigned":/experiments."enrolled": columns (Instrumentation) for group assignment values and I will run KR retention analysis manually.

HNordeenWMF updated the task description. (Show Details)Jul 2 2026, 8:47 PM

HNordeenWMF raised the priority of this task from Low to Medium.Jul 2 2026, 10:13 PM

HNordeenWMF lowered the priority of this task from Medium to Low.Jul 7 2026, 7:38 PM

HNordeenWMF raised the priority of this task from Low to Medium.Jul 13 2026, 2:46 PM

HNordeenWMF moved this task from P4 (4 sprints out) to P2 (Sprint After Next) on the Explore Feed Refresh iOS board.Jul 13 2026, 7:20 PM

HNordeenWMF updated the task description. (Show Details)Jul 13 2026, 10:17 PM

HNordeenWMF updated the task description. (Show Details)Jul 13 2026, 10:22 PM

HNordeenWMF raised the priority of this task from Medium to High.Jul 13 2026, 11:09 PM

HNordeenWMF moved this task from Backlog to P2 (Needs refinement) on the App Experience board.

HNordeenWMF mentioned this in T432074: Scale iOS Home Feed to remaining 50% of users.Jul 13 2026, 11:15 PM

Mazevedo set the point value for this task to 5.Jul 15 2026, 2:18 PM

HNordeenWMF moved this task from P2 (Needs refinement) to P1 (Ready for sprint) on the App Experience board.Jul 21 2026, 4:26 PM

HNordeenWMF moved this task from P2 (Sprint After Next) to P1 (Next Sprint) on the Explore Feed Refresh iOS board.Jul 30 2026, 2:24 PM

Mazevedo updated the task description. (Show Details)Jul 30 2026, 5:04 PM

Mazevedo changed the point value for this task from 5 to 3.

HNordeenWMF updated the task description. (Show Details)Wed, Aug 12, 3:52 PM

HNordeenWMF moved this task from P1 (Ready for sprint) to AppEx Sprint [FY2627 Aug 12 - Aug 25] on the App Experience board.Wed, Aug 12, 4:34 PM

HNordeenWMF edited projects, added: App Experience (AppEx Sprint [FY2627 Aug 12 - Aug 25]); removed: App Experience.

Tsevener updated the task description. (Show Details)Wed, Aug 12, 7:55 PM

GOlson-WMF moved this task from Ready to Doing on the App Experience (AppEx Sprint [FY2627 Aug 12 - Aug 25]) board.Fri, Aug 14, 7:13 PM

GOlson-WMF claimed this task.Fri, Aug 14, 7:16 PM

HNordeenWMF moved this task from P1 (Next Sprint) to P0 (Current Sprint) on the Explore Feed Refresh iOS board.Mon, Aug 17, 5:29 PM

@GOlson-WMF is there an associated PR with this ticket?

Test plan:

To test updating from old:
Fresh install on MAIN branch (or prior version) OR have prior version installed
Not deleting old version, update to new one (run new branch)
You will be assigned either control or groupB
If you are assigned control:

Make sure you see explore as usual, no new onboarding

If you are assigned groupB:

Make sure you see the new one time onboarding and are able to go through it
Make sure you don’t see it on subsequent installs
Make sure you see Community and For You

To test fresh install:
New device/fresh install (fully delete any old versions)
If you're assigned control:

Make sure you go through the legacy onboarding
Make sure you see explore
Make sure you don't see for you/ community

If you're assigned group B

Make sure you see the NEW onboarding
Make sure you do NOT see the one time onboarding
Make sure you see for you/ community / etc

HNordeenWMF renamed this task from [M/L] Deploy Explore feed revamp as A/B Test to [M/L] Deploy Onboarding & "For you" feed revamp as A/B Test.Tue, Aug 18, 5:23 PM

https://github.com/wikimedia/wikipedia-ios/pull/6076

Added draft PR for now!

GOlson-WMF moved this task from Doing to Needs Code Review on the App Experience (AppEx Sprint [FY2627 Aug 12 - Aug 25]) board.Tue, Aug 18, 8:43 PM

HNordeenWMF mentioned this in T435333: [Feature Release Review] iOS Home Feed Phase 1 Release (Onboarding, For you, Settings).Wed, Aug 19, 1:47 PM

GOlson-WMF mentioned this in T427577: Existing User Re-onboarding — New "Home" Feed Introduction.Fri, Aug 21, 5:32 PM

tonisevener closed https://github.com/wikimedia/wikipedia-ios/pull/6076

Available in: TestFlight Wikipedia 2026.08.21 (6153)

GitHubPRBot mentioned this in T435256: Release 2026.08.28.Fri, Aug 21, 9:08 PM

Tsevener moved this task from Needs Code Review to Needs QA* on the App Experience (AppEx Sprint [FY2627 Aug 12 - Aug 25]) board.Fri, Aug 21, 9:19 PM

Tsevener updated the task description. (Show Details)Mon, Aug 24, 10:09 PM

Tsevener updated the task description. (Show Details)

Testing on 6154-beta-2026-08-21
Tested on:
iPhone 16, iOS 26.6
iPhone 11, iOS 26.5
iPad Air, iPadOS 26.6

Requirement	✅❓❌	Notes
The release with the Explore Feed revamp should only be made availalbe to 50% of iOS app users
A: Control (Explore tab, no changes to feed)
Existing onboarding	✅	Works as expected
B: Variant with redesigned feed
Renamed to Home	✅	Works as expected
Updated onboarding with Interest selection for new installs and existing app users	✅	Works as expected
“For you” feed with new language selector	✅	Works as expected
Additional settings for "For you"	✅	Works as expected
Existing Explore Feed wrapped into “Community”, With any modules that have been added to "For you" removed	✅	Works as expected
Audience: All languages, logged-in and logged-out	✅	Works as expected
Analytics must be able to measure retention to the app for Group A and Group B	❓	Should be checked by @SNowick_WMF
Test should have no end-date. We will create a scaling task to end the test.	❓	Assuming this is good, I can't really check it
Note: follow-up releases will contain
Additional “For you” modules (Random, Places, Games), removing from legacy feed as we go	❌	Not present
Rewritten “Community” tab	❌	Not completed at this time

mpopov mentioned this in T434941: [DE3.2.8] Experiment exposure logging in mobile apps.Thu, Aug 27, 10:07 PM

Seddon closed this task as Resolved.Tue, Sep 1, 5:09 PM

[M/L] Deploy Onboarding & "For you" feed revamp as A/B TestClosed, ResolvedPublic3 Estimated Story PointsActions