Page MenuHomePhabricator

Quick Analysis: Thank You Pages: custom account creation pages for sv, it, ja, fr, nl
Closed, ResolvedPublic

Assigned To
Authored By
KStoller-WMF
Mar 8 2023, 12:45 AM
Referenced Files
F36932358: Screen Shot 2023-03-29 at 10.12.03.png
Mar 29 2023, 5:50 PM
Unknown Object (File)
Mar 21 2023, 11:20 PM
Unknown Object (File)
Mar 21 2023, 11:19 PM

Description

User story:
As the Growth Product Manager I want to check instrumentation and account creation to be sure this release is being monitored and having the impact that we expect.


Objective:

As part of WMF's work around the Newcomer Experience Pilot, we have this objective: Account creation: internet users in our target markets learn that they can contribute to Wikipedia, are prompted to begin contributing and to create an account to have access to the newcomer experience.

Background:
Last year, the Newcomer Experience Pilot project tested several different ways to increase account creation and activation:
https://meta.wikimedia.org/wiki/Research:Newcomer_Experience_Pilot_Project-_Thank_You_Pages_and_Thank_You_Banners

The Thank you page experiment was only conducted in a few regions / languages, but showed promise:
~29,000 Registrations, ~1,300 Constructive Activations (defined as editing within 24 hours of registration and that edit not being reverted within 48 hours). We are now expanding this experiment to more languages and regions.

Acceptance Criteria:

Given the edit / create account call to action is added to Thank You pages,
Then one week after launch, we quickly check data to make sure instrumentation is working as expected and the account creation metrics look healthy

Details

Due Date
Mar 24 2023, 7:00 AM

Event Timeline

KStoller-WMF created this task.
KStoller-WMF set Due Date to Mar 24 2023, 7:00 AM.

hi @nettrom_WMF , hope you're well! Joseph made an updated version of our TY Page donation dashboard, here is a screenshot.
Let me when/if you'd like me to share data from any particular time-range to help make sense of the account data. As you can see, Japan and Sweden have ongoing Fundraising campaigns right now so their "TY Page views" are much higher!

Thanks for the update, @HNordeenWMF!

I'm only seeing the file number, not the image. I think because it's "restricted" to only show to you?
I think you can update the sharing preferences on the right Phab column under "Referenced Files".

HNordeenWMF attached a referenced file: Unknown Object (File). (Show Details)Mar 22 2023, 5:58 PM

ah thank you @KStoller-WMF! I think it should be attached now.

@HNordeenWMF : I'm currently working on gathering these statistics and noticed that T321370 specifies two languages for each campaign: the local language of the wiki as well as English. The language is then reflected in the uselang parameter in the URL. When grabbing the data I noticed this parameter doesn't show up (except for a few test visits), and from playing around with the Thank You page I did not see the uselang parameter in the link to account creation. I'm not sure if this is a bug or not?

@nettrom_WMF - My sense is that we did more in-depth analysis for the initial experiment, and at this point it should be OK to simplify and just look at combined metrics.
But if you or @HNordeenWMF think splitting the data by language is definitely needed, feel free to speak up.

@nettrom_WMF - My sense is that we did more in-depth analysis for the initial experiment, and at this point it should be OK to simplify and just look at combined metrics.
But if you or @HNordeenWMF think splitting the data by language is definitely needed, feel free to speak up.

When it comes to this analysis we'll have to look at the combined metrics since there's no language data. I wanted to make sure that @HNordeenWMF is aware of this though, as I perceived it as a discrepancy between the specifications and deployment. The Thank You pages themselves have two separate language versions, as reflected in the dashboard screenshots shown above.

@nettrom_WMF I don't think it's an issue for us, but @Pcoombe might have something to add too as he understands the TY Page much better than I do.

To make sure I understand your point "we'll have to look at the combined metrics since there's no language data". Does this mean we won't be able to know for example English-France registrations, we'll only be able to know total France registrations (en, fr) as a portion of total France TY Page views? That seems ok.
I'm assuming we'd still be able to look at total English registrations (all countries) as a portion of total English TY Page views (all countries) too? Or do we also lose that ability.

Fundraising is very country-based in our goals, whereas I know other teams are language-wiki based. If it would be more beneficial for Product to have specific language-wiki data within each country, we could all work to update the links.

To make sure I understand your point "we'll have to look at the combined metrics since there's no language data". Does this mean we won't be able to know for example English-France registrations, we'll only be able to know total France registrations (en, fr) as a portion of total France TY Page views? That seems ok.
I'm assuming we'd still be able to look at total English registrations (all countries) as a portion of total English TY Page views (all countries) too? Or do we also lose that ability.

"Yes" to your first question. I'm unsure about the answer to the second question. Maybe if I describe my understanding of the workflow we can figure this out? Here's a diagram I drew based on how T321370 defines the landing page URLs based on donations from Sweden:

Screen Shot 2023-03-29 at 10.12.03.png (1×2 px, 246 KB)

I'm not sure if my assumption that donations in English originate from the English Wikipedia is correct? Would they be from either Swedish or English Wikipedia, but follow the language of the user (e.g. if I have my interface language set to "English" on Swedish Wikipedia)?

Since the uselang parameter is not present in the URLs, the links from the Thank You pages all go to Swedish Wikipedia and show the account creation page in Swedish. We're able to identify the views of the account creation page from the campaign parameter, and then identify account creations based on the same parameter.

If what we want is to be able to distinguish country & language all the way through, having campaign parameters that define both (e.g. as done in T305659) would enable the most reliable and easily retrievable statistics. Having uselang in the URL parameters enables us to understand views of the account creation page by language, but we then lose that distinction at account creation. Another possibility is to geolocate the landing page views and account creations, something that we haven't done previously.

nettrom_WMF subscribed.

Editing this to remove the posted data because the more recent table posted below is the authoritative one.

Thanks for this! Apologies for the delay in responding here.

Those seeing the Thank You page in English do get a link to create an account on the English Wikipedia e.g. in Sweden the English language Thank You page links to https://en.wikipedia.org/w/index.php?title=Special:CreateAccount&campaign=typage-nwe-sv-2023.

This seemed to make more sense for the user, and is consistent with what we previously did in Latin America. I mentioned this in Slack, but I don't think the links in T321370 got updated sorry.

I'm not sure if my assumption that donations in English originate from the English Wikipedia is correct? Would they be from either Swedish or English Wikipedia, but follow the language of the user (e.g. if I have my interface language set to "English" on Swedish Wikipedia)?

We only show fundraising banners to logged-out users, who can't set an interface language. So yes, donations in English will always be originally from the English Wikipedia (there may be a very small number who used the sidebar donate link on a different English project e.g. English Wiktionary)

In any case I think we're still okay just splitting by campaign parameter / country unless Product particularly wants to look at language splits. @HNordeenWMF what do you think?

nettrom_WMF changed the task status from Open to In Progress.Apr 5 2023, 8:44 PM
nettrom_WMF claimed this task.

Those seeing the Thank You page in English do get a link to create an account on the English Wikipedia e.g. in Sweden the English language Thank You page links to https://en.wikipedia.org/w/index.php?title=Special:CreateAccount&campaign=typage-nwe-sv-2023.

This seemed to make more sense for the user, and is consistent with what we previously did in Latin America. I mentioned this in Slack, but I don't think the links in T321370 got updated sorry.

I thought I'd checked the English language Thank You page as well, but somehow didn't see it point to English Wikipedia. We're missing all the data from that wiki, so I'm reassigning this to me while I gather that data and update the table with it.

Thanks @Pcoombe, agreed I think the current parameters are fine with us.
Earlier on I noticed the links being the same for en+ other language, and @KStoller-WMF said product didn't require that breakdown by language within country.

nettrom_WMF closed this task as Resolved.EditedApr 11 2023, 8:31 PM

I've gathered all the data from English Wikipedia and in the process fixed a bug in our processing so that we also correctly count mobile page views after the desktop to mobile redirect. We do not report data for the French and Dutch campaigns because of low numbers.

I'm also not reporting any specific numbers for English because the activation numbers are too low. The registration proportions are on part with other languages with a range of 30–60% of the unique visitors. Activation varies greatly and for some campaigns comes in at 0%. Key takeaway: folks appear to be comfortable registering in English but not comfortable contributing.

LanguagePlatformPage viewsUnique visitorsRegistrationsRegistration %ActivationsActivation %
ItalianDesktop64254325847.5%3212.4%
Mobile31028111039.1%1210.9%
JapaneseDesktop3,1702,6171,18245.2%504.2%
Mobile2,6532,39697840.8%414.2%
SwedishDesktop1571436847.6%913.2%
Mobile1441346649.3%913.6%

Methodological notes: Data collected spans 2023-03-07 to 2023-03-26. "Unique visitors" is counted using a combination of IP address and User-Agent to get more correct counts in situations where users share IP addresses. Previous analysis only used IP addresses as a shortcut. This affects some wikis more than others (e.g. Japanese sees a 2–3% increase in unique visitors). The count of registrations excludes known test accounts and Wikipedia app registrations, which is consistent with our previous analyses. "Constructive activation" means making at least one non-reverted edit within 24 hours of registration.

Based on my discussions with @KStoller-WMF this work is complete and so I'm resolving this task. Feel free to ping me on Slack with follow up questions.

Edited on 2023-08-29: the notebook for this analysis is now on gitlab: T331495-TY-page-leading-indicator.ipynb