Can this be done with old-fashioned user testing, and not a live A/B test in the app?

@Dbrant

Can this be done with old-fashioned user testing, and not a live A/B test in the app?

No, A/B testing will give us more insights. A usability test with 5-10 users where we ask about preference won’t be representative for a base of 5 million users. Real usage data from real Wikipedia Android users is needed to make a decision. This change could significantly contribute to making the app easier to use (and keeping users engageed for longer). As this change in terminology will be considered for iOS, it’s better to have data evidence.

Notes for testing:

• Go to Developer Settings and scroll all the way down to "Whether the user sees a Home icon...". Change this setting to an even number to get in the A group, or an odd number to get in the B group.

Notes for analytics:

• This shall use the existing MobileWikiAppABTest schema with the test groups of exploreVsHome_GroupA and exploreVsHome_GroupB.

Thanks @Dbrant, should the event data for Explore and Home be in the MobileWikiAppNavMenu table? Not seeing any events for Home or Explore. I did a SELECT distinct(event.menuItem) FROM MobileWikiAppNavMenu query and got the following:

Recent
More
Today
Login
SavedPages
Random
Nearby
ReadingLists
Feed
Unknown
Main Page

This will need to be instrumented or am I looking in the wrong place.

@Dbrant → changed it to an even number in the latest Alpha but I’m still seeing Explore. Any idea why?

In T260332#6398510, @SNowick_WMF wrote:

Thanks @Dbrant, should the event data for Explore and Home be in the MobileWikiAppNavMenu table? Not seeing any events for Home or Explore. I did a SELECT distinct(event.menuItem) FROM MobileWikiAppNavMenu query and got the following:
This will need to be instrumented or am I looking in the wrong place.

@SNowick_WMF The MobileWikiAppNavMenu is a very old schema, and is no longer used. It was used when we had a navigation menu (the menu that pulls out from the left edge). I believe the question of session length can be answered by using the MobileWikiAppSessions table, combined with the A/B test groups I mentioned above. (unless you can think of a more correct way to do it.)

In T260332#6398864, @schoenbaechler wrote:

@Dbrant → changed it to an even number in the latest Alpha but I’m still seeing Explore. Any idea why?

@schoenbaechler
Hang on, I thought this had to do with changing the icon in the Page overflow menu:

...whereby "Home" refers to the Main screen, which *contains* the Explore feed. Are we saying that the Explore feed is now "Home"? This would be a fairly major change, since we use the "Explore" branding in a lot of other places, including our Play Store description.

@Dbrant

Hang on, I thought this had to do with changing the icon in the Page overflow menu:

Oh I see, one of these rare misunderstandings. According to the discussion in T256819#6373239, we wanted to change it for the main nav, see conversation below 👇:

Hey @Charlotte, thought about the points we discussed at our 1:1 on Friday.

Use house icon and Home terminology instead of world icon and Explore label in article overflow menu

This sounded like a good idea to me before the weekend. I gave it some more thought this morning and am hesitant to introduce one more notion to the app’s current concept. When we want to introduce new terminology like Home instead of Explore, it needs to be done on an app(s) level. Definitely worth considering renaming Explore to Home on both iOS and Android - but it does not make sense to do it just in one menu. Should we create a separate task for this?

You're right that it doesn't make sense to do it on one menu. My hypothesis is that many of our users don't know what "Explore" would mean in this context, however. (It even took me a minute to figure out that it wasn't an article tool, but rather a link to the feed.) I would suggest creating a ticket to assess whether this hypothesis is actually correct.

Are we saying that the Explore feed is now "Home"?

No, we’re A/B testing a possible rebrand. Running an A/B test with just the changes in the article overflow menu won’t give us "enough leverage" to decide if we want to brand it as Home or Explore in the future (as it could affect other platforms) . To me, we need to A/B test in the main nav first. Then it’ll give us data on a possible, app-wide, rebrand.

This would be a fairly major change, since we use the "Explore" branding in a lot of other places, including our Play Store description.

YES to bold changes if the result is. more clarity for users and more engagement, the efforts will be worth it. And I don’t think it’s an effort to change / update the marketing material for it.

In T260332#6401903, @schoenbaechler wrote:

@Dbrant

Are we saying that the Explore feed is now "Home"?

No, we’re A/B testing a possible rebrand. Running an A/B test with just the changes in the article overflow menu won’t give us "enough leverage" to decide if we want to brand it as Home or Explore in the future (as it could affect other platforms) . To me, we need to A/B test in the main nav first. Then it’ll give us data on a possible, app-wide, rebrand.

This would be a fairly major change, since we use the "Explore" branding in a lot of other places, including our Play Store description.

YES to bold changes if the result is. more clarity for users and more engagement, the efforts will be worth it. And I don’t think it’s an effort to change / update the marketing material for it.

But by all means leave the A/B test on in the article overflow menu as well @Dbrant, since I'm eager to see if it actually makes a difference. We may want to make the change in the overflow menu irrespective of what we do in the app as a whole.

It's going to be a slightly bigger change than you're probably thinking of @schoenbaechler to update the material in all the languages we currently have in the Play Store, in our technical documentation, etc etc. We'll probably still be finding mentions of "Explore" 6 months from now, if we decide to make the change. So the uptick in engagement as shown in the test will have to be significant. But if it is, then you're quite right that it will be worth it.

Tested the A and B groups and I am seeing the expected icons. Looks good on 2.7.50330-alpha-2020-08-25

Looking at avg. session count and avg. session length between both groups shows no appreciable difference. Data for 30 day period:

Since these user groups are quite large I took a smaller cohort of 200 users per group with the same join date to make sure any differences weren't getting diluted and results were the same.

If we want to measure user preference (ie do more users choose and use Home vs Explore more frequently) we will have to use event counts. Let's discuss a framework to AB test this further in a way that gives us more data to work with.

@SNowick_WMF

Looking at avg. session count and avg. session length between both groups shows no appreciable difference. Data for 30 day period:

Can you define what Group A and Group B is?

If we want to measure user preference (ie do more users choose and use Home vs Explore more frequently) we will have to use event counts. Let's discuss a framework to AB test this further in a way that gives us more data to work with.

Yes, please!

I believe Group A sees Home and Group B sees Explore, @Dbrant can verify if this is correct.

For further testing let's add it to @JTannerWMF's planning for next quarter.

I would consider this A/B test complete, with no statistically significant difference in behavior between the groups. This means it's basically our choice to keep "Explore", or change it to "Home" at this point.
@schoenbaechler Would you like to make the final call?

Thx @Dbrant – I’ll discuss this with Carolyn tomorrow, July 5. Tendency I’d say at this point is to keep Explore...

Talked to @cmadeo – we agreed to stick with “Explore” for now! Thanks all folks involved for performing this A/B test.