Page MenuHomePhabricator

Homepage: discovery measurement (Feb 2020)
Closed, ResolvedPublic

Description

Here is Feb 2020, we want to remeasure the rate at which newcomers are discovering the homepage because we now want more newcomers to begin the newcomer tasks workflow. We're trying to decide whether the best way is to get more newcomers on their homepage, or to increase the affordance of the workflow once the newcomers are on their homepage.

Since we previously measured this in September 2019 via T228338, we have added/changed some of the discovery features. We also want additional breakdowns that we didn't previously look at. Here's what we want to know (but @nettrom_WMF may add other aspects if he realizes they would also be illuminating):

Percentage of newcomers visiting their homepage broken out by...

  • wiki
  • desktop/mobile platform (since some percentage of users use both platforms, perhaps this could be distinguished by the platform on which the account was created)

We want to break this down by wiki and platform at the same time.

We also want an overview of the paths taken to the Homepage on the first visit, again broken down by wiki and platform, like we did in the previous analysis.

We should include these wikis in the analysis:

  • Czech
  • Korean
  • Arabic
  • Vietnamese
  • Hungarian
  • Ukrainian
  • Armenian

Data from recent weeks/months is sufficient -- we're not looking for change over time. We just want to know what discovery rates are these days.

Event Timeline

The first pass of this analysis is now completed. In the original analysis in T228338 we only studied two Wikipedias, Czech and Korean, because those were the ones where the Homepage discovery features were deployed to. The new one has seven Wikipedias. We also want results split up by whether the user registered and/or visited on the desktop or mobile platform. This means we have 14 graphs. Instead of putting all of them in the same image, I chose to split this up by wiki.

I gathered data for all wikis for the first three months of 2020. The criteria were the same as in the previous analysis, meaning all registrations on the given wikis (thus no auto-created accounts), and excluding known test accounts, API registrations (these are mainly app accounts), users who don't have the Homepage enabled (we have 20% control groups on our four target wikis), and users who turned the Homepage on/off (self-selection). Data only counts the first visit a user makes to the Homepage, and we give every user a 48 hour window after registration to make their first visit. This also means that the last date of data is 2020-03-29.

Before making any plots of tables, I investigated what proportion of users make their first visit to the Homepage on a different platform than the one they registered on. If this proportion is very low, it would mean that we can exclude these users and thus reduce the number of categories we deal with. Across the dataset, 1.3% of users (314 out of 23,962) made that switch. I found this to be low enough that they could be excluded.

Visit Proportion Across Time:

Similarly as in the previous analysis, I made graphs for the proportion of registrations who visited the Homepage counted on a daily basis, split by platform. The date is the date of registration. A 7-day moving average is added to smooth out the variation. Wikis are listed in order of their database code (e.g. Armenian is fourth because the code is "hywiki"), and the same order is used in the subsequent tables below.

Arabic:

Czech:

Hungarian:

Armenian:

Korean:

Ukrainian:

Vietnamese:

Monthly overall proportion:

Next, I calculated the overall proportion for February and March, split by wiki and platform. I chose to hold January out because we deployed to three additional wikis in that month, or in other words, I only used months where we have (almost) complete data. In this case, The proportions were largely similar for both February and March, and I'm only reporting the March numbers because that month is the most recent. This results in the following table:

WikiPlatformN RegsN VisitsVisit %
ArabicDesktop2,2421,57870.4
ArabicMobile4,3082,85266.2
CzechDesktop78054870.3
CzechMobile34420359.0
HungarianDesktop47823348.7
HungarianMobile2638833.5
ArmenianDesktop1006565.0
ArmenianMobile673349.3
KoreanDesktop1,07853249.4
KoreanMobile79734343.0
UkranianDesktop70640957.9
UkranianMobile39415840.1
VietnameseDesktop1,54368244.2
VietnameseMobile1,12334030.3

Routes to the Homepage:

Lastly, we look at the routes users took to get to the Homepage, based on the referer_route field in the HomepageVisit schema. In this case we also want to split the data by platform, but choose to use the is_mobile field in the schema rather than where the user registered so we can use all available data. In other words, the 1.3% of users who switch platforms are included. The (rather large) table below shows these proportions for the month of March:

WikiPlatformRoute to HomepageN VisitorsPercent
ArabicDesktopspecialwelcomesurvey1,01262.4
ArabicDesktoppersonaltoolslink36722.6
ArabicDesktopspecialconfirmemail18011.1
ArabicDesktopspecialcontributions261.6
ArabicDesktopusertalkpagetab191.2
ArabicDesktopuserpagetab181.1
ArabicMobilespecialwelcomesurvey1,84964.0
ArabicMobilepersonaltoolslink65522.7
ArabicMobilespecialconfirmemail1766.1
ArabicMobilespecialcontributions1324.6
ArabicMobileuserpagetab491.7
ArabicMobileusertalkpagetab270.9
CzechDesktopspecialwelcomesurvey28550.9
CzechDesktoppersonaltoolslink13824.6
CzechDesktopspecialconfirmemail10719.1
CzechDesktopspecialcontributions213.8
CzechDesktopuserpagetab61.1
CzechDesktopusertalkpagetab30.5
CzechMobilespecialwelcomesurvey13565.9
CzechMobilepersonaltoolslink4019.5
CzechMobilespecialconfirmemail167.8
CzechMobilespecialcontributions125.9
CzechMobileuserpagetab10.5
CzechMobileusertalkpagetab10.5
HungarianDesktoppersonaltoolslink12551.0
HungarianDesktopspecialconfirmemail10241.6
HungarianDesktopuserpagetab93.7
HungarianDesktopusertalkpagetab52.0
HungarianDesktopspecialcontributions41.6
HungarianMobilepersonaltoolslink3941.9
HungarianMobilespecialconfirmemail3335.5
HungarianMobilespecialcontributions1415.1
HungarianMobileuserpagetab77.5
ArmenianDesktoppersonaltoolslink5480.6
ArmenianDesktopspecialconfirmemail69.0
ArmenianDesktopuserpagetab34.5
ArmenianDesktopspecialcontributions23.0
ArmenianDesktopother11.5
ArmenianDesktopusertalkpagetab11.5
ArmenianMobilespecialcontributions1541.7
ArmenianMobilepersonaltoolslink1336.1
ArmenianMobilespecialconfirmemail616.7
ArmenianMobileuserpagetab12.8
ArmenianMobileusertalkpagetab12.8
KoreanDesktopspecialwelcomesurvey32660.0
KoreanDesktoppersonaltoolslink13524.9
KoreanDesktopspecialconfirmemail6011.0
KoreanDesktopuserpagetab81.5
KoreanDesktopusertalkpagetab81.5
KoreanDesktopspecialcontributions61.1
KoreanMobilespecialwelcomesurvey20658.4
KoreanMobilepersonaltoolslink10630.0
KoreanMobileuserpagetab133.7
KoreanMobileusertalkpagetab133.7
KoreanMobilespecialcontributions82.3
KoreanMobilespecialconfirmemail72.0
UkranianDesktoppersonaltoolslink19945.7
UkranianDesktopspecialconfirmemail18041.4
UkranianDesktopuserpagetab235.3
UkranianDesktopspecialcontributions204.6
UkranianDesktopusertalkpagetab133.0
UkranianMobilepersonaltoolslink7947.0
UkranianMobilespecialconfirmemail3621.4
UkranianMobilespecialcontributions3520.8
UkranianMobileuserpagetab137.7
UkranianMobileusertalkpagetab53.0
VietnameseDesktoppersonaltoolslink37753.2
VietnameseDesktopspecialconfirmemail19627.7
VietnameseDesktopspecialcontributions10014.1
VietnameseDesktopusertalkpagetab223.1
VietnameseDesktopuserpagetab131.8
VietnameseMobilepersonaltoolslink17950.6
VietnameseMobilespecialcontributions10228.8
VietnameseMobilespecialconfirmemail5515.5
VietnameseMobileuserpagetab144.0
VietnameseMobileusertalkpagetab41.1

While the task description doesn't ask questions, I think one of the patterns here is that we get a lot of visits through the Welcome Survey on the wikis where it's deployed. Thus we might want to consider whether the survey should be regarded as a key element in the newcomer experience.

Moving the task over to @MMiller_WMF's column for review.

MMiller_WMF closed this task as Resolved.EditedApr 1 2020, 11:59 PM

This is great, @nettrom_WMF. Thank you! Here are the main points that I'm taking away from this:

  • Seeing that some wikis, like Arabic and Czech, have two thirds of newcomers visiting their homepage, we know that the discovery features we've built have the potential to drive newcomers to their homepage. In other words, newcomers are definitely presented with plenty of opportunities to visit.
  • But it's important to note that not all wikis are visiting at those high rates. Vietnamese and Hungarian, for instance, are visiting at more like 40%. This gap shows that more can be done to increase discovery.
  • Looking at the routes to the homepage, we see that:
    • For wikis that have it, the welcome survey is the dominant path, on both desktop and mobile. It's responsible for 50-65% of homepage discovery
    • On wikis without the survey, users instead tend to discovery their homepage through their personal tools link (presumably prompted by the guided tours) or through email confirmation. We didn't plan this, but the absence of the survey here has really given us some insight into how homepage discovery works.
    • On mobile, the Special:Contributions button also leads to a lot of discovery. I'm not surprised to see that, because I think that the idea for this route was prompted by user research or analysis showing that many mobile users think that the "Contributions" menu option is how they should contribute.
  • So, seeing all this, the most obvious explanation for the gap between Arabic/Czech and the other wikis is the welcome survey, which is only present on Arabic, Czech, and Korean. Weirdly, though, Korean is not up there with Arabic and Czech in terms of high discovery rates, even though it has the welcome survey, and the welcome survey route is being used there at the same rates as those two wikis.
  • We are in the process of deploying the welcome survey to the other wikis, but we should remember that homepage discovery is not a good enough reason to have the welcome survey. In other words, we could probably have the same effect with an account creation landing page that doesn't ask any survey questions -- or certainly by just redirecting users to their homepage after account creation, or by replacing Main Page with the homepage. Perhaps we should consider that more heavy-handed idea.
  • Mobile discovery is lower than desktop discovery. There may be improvements to make there.

Below are graphs that I made from the numbers: