Page MenuHomePhabricator

Welcome emails: opt-in checkbox
Closed, ResolvedPublicFeature

Assigned To
Authored By
EdErhart-WMF
Mar 8 2022, 2:23 AM
Tokens
"Love" token, awarded by TheresNoTime.

Description

Feature summary:

Add a checkbox to the Special:WelcomeSurvey that would give us permission to send a celebratory welcome email with some tips and tricks for new users. We believe that sending an email like this would, if expanded, improve the activation and retention rate of new user accounts. Details follow below:

Main text

"I want to receive tips via email on how to start editing."

Disclaimer

Spanish: "Tenga presente que la lista de correo se lleva a cabo con la tecnología de MailChimp, lo que significa que el uso que hace MailChimp de su información se rige por su [https://www.intuit.com/privacy/statement/es/ Política de privacidad] y sus [https://mailchimp.com/es/legal/terms/ Condiciones de servicio]."

(English: Please note that this mailing list is powered by MailChimp, which means MailChimp’s use of your information is governed by their [https://www.intuit.com/privacy/statement/ Privacy Policy] and [https://mailchimp.com/legal/terms/ Terms of Service].)

We should put put the MailChimp disclaimer under the checkbox in gray, the way it currently has the "We noticed you didn't..." text under the email field.

Location

  • It should be the last item on the welcome survey.
  • The checkbox should appear for all users regardless of whether they have entered an email address (this requirement is for simplicity, but we would ideally want it to appear only if the user has entered an email.)

Wikis needed

Spanish Wikipedia. Any visitor can view it, it is not contingent on being part of a campaign.

Platforms

Desktop and mobile

Date

The week of 4 April [pushed back]

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Hi @MMiller_WMF and @EdErhart-WMF , I wanted to clarify that this extra checkbox is not going to be on the sign-up page, but on the welcome survey as we discussed.
https://test.wikipedia.org/wiki/Special:WelcomeSurvey

Screenshot of current Welcome Survey for reference:

image.png (1×3 px, 578 KB)

Hi all, thanks for the feedback here. I've added what I believe to be all the requested information into the task description. @RHo, you're correct that we're requesting that it go onto the welcome survey. I inadvertently used "sign-up page" as shorthand.

@EdErhart-WMF -- thanks for filling in more details.

Welcome survey

  • Though this might not matter once translated to Spanish, I think a less ambiguous way to write the checkbox would be: "I want to receive tips via email on how to start editing."
  • I think we should only show the checkbox only if the user has an email address entered either on the account creation page, or if they put it in the field on the welcome survey. If they have not entered an email, they should not have the checkbox. Does this work for you?
  • I think we should put put the MailChimp disclaimer under the checkbox in gray, the way it currently has the "We noticed you didn't..." text under the email field. Does this work for you?

Exporting the list

  • Which users will you want to receive your emails? All new accounts? Only those that have edits?
  • Should we export all users with email addresses? Or will they need to have confirmed email addresses, meaning they received the confirmation email and clicked the link?
  • When we export lists for MailChimp, what format will you need? What fields will you need?
MMiller_WMF renamed this task from Opt-in checkbox to receive experimental welcome email to Welcome emails: opt-in checkbox.Mar 14 2022, 10:49 PM

Moving back to "Needs Design" as we await answers from @EdErhart-WMF.

Hey @MMiller_WMF,

For the welcome survey items:

  • I've edited the task description to reflect that wording. Thank you!
  • Both of those proposals sound great. I agree that having a checkbox there without an email is not helpful to anyone. :-)

For the exporting:

  • We'd like to send emails to all new accounts that have addresses regardless of edits, and we are writing our copy in a way to account for both kinds of users.
  • Great question. I would prefer to send an email to all of them regardless of confirmation; we can accept that some of those addresses will be invalid. However, I'm open to changing course if this is a bad idea for reasons I haven't seen!
  • We need the email addresses in a CSV file organized into a single column. MailChimp has some info on this as well.

@EdErhart-WMF this checkbox will display to all users who visit eswiki and want to create a new account, is that correct? Or will it only show for users who are using a specific URL to create an account (i.e. as part of a campaign)?

@EdErhart-WMF this checkbox will display to all users who visit eswiki and want to create a new account, is that correct? Or will it only show for users who are using a specific URL to create an account (i.e. as part of a campaign)?

@MMiller_WMF answered this earlier with: all eswiki visitors to Special:WelcomeSurvey would see this, if they have entered an email on Special:CreateAccount

Change 771057 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):

[mediawiki/extensions/GrowthExperiments@master] WelcomeSurvey: Add checkbox for opt-in to marketing emails

https://gerrit.wikimedia.org/r/771057

@kostajh -- I made an edit to the task description to ensure that all specifications are there. The one material thing I changed is: "The checkbox only appears if the user entered an email address on Special:CreateAccount or in the welcome survey." I just want to make sure we're covered there.

@EdErhart-WMF -- I'm going to restart our discussion about exports over on this task: T303780: Welcome emails: export opted-in users.

Also, @EdErhart-WMF, please take a look at @kostajh's screenshots above to see if anything looks different than you would want.

My thanks to you both! @kostajh, those screenshots look great overall. I'm assuming there will still be a place for a user to enter their email address? :-)

My thanks to you both! @kostajh, those screenshots look great overall. I'm assuming there will still be a place for a user to enter their email address? :-)

Right, I'll fix that up.

Test wiki created on Patch demo by KHarlan (WMF) using patch(es) linked to this task:
https://patchdemo.wmflabs.org/wikis/fa384c3ada/w/

Test wiki created on Patch demo by KHarlan (WMF) using patch(es) linked to this task:
https://patchdemo.wmflabs.org/wikis/fa384c3ada/w/

@EdErhart-WMF @MMiller_WMF you can check out the link above and create an account to see how the checkbox placement looks. Thanks!

Change 771057 merged by jenkins-bot:

[mediawiki/extensions/GrowthExperiments@master] WelcomeSurvey: Add checkbox for opt-in to marketing emails

https://gerrit.wikimedia.org/r/771057

Change 771451 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):

[operations/mediawiki-config@master] GrowthExperiments: Set GEWelcomeSurveyShowMailingListQuestion

https://gerrit.wikimedia.org/r/771451

Change 771451 merged by jenkins-bot:

[operations/mediawiki-config@master] GrowthExperiments: Set GEWelcomeSurveyShowMailingListQuestion

https://gerrit.wikimedia.org/r/771451

Mentioned in SAL (#wikimedia-operations) [2022-03-16T20:40:20Z] <urbanecm@deploy1002> Synchronized wmf-config/InitialiseSettings.php: [no-op] 8efa537: GrowthExperiments: Set GEWelcomeSurveyShowMailingListQuestion (T303240) (duration: 00m 53s)

@kostajh -- thanks; I just tried out the patchdemo a few different ways. The one thing that I can see that would be an improvement is to only show the checkbox if the user has entered an email address, either on Special:CreateAccount or in the field on Special:WelcomeSurvey. Do you see a simple way to do that? The rule could be as simple as "Appear if the user entered an email address on Special:CreateAccount OR has typed anything in the email box on Special:WelcomeSurvey".

If there is not a simple way to do it, then I don't think the consequences would be great if we skipped this nuance.

I've just tried it out! Thanks, @kostajh. My only feedback mirrors Marshall's.

Test wiki created on Patch demo by KHarlan (WMF) using patch(es) linked to this task:
https://patchdemo.wmflabs.org/wikis/fa384c3ada/w/

@EdErhart-WMF @MMiller_WMF you can check out the link above and create an account to see how the checkbox placement looks. Thanks!

You can also see it here: https://es.wikipedia.beta.wmflabs.org/wiki/Especial:Encuesta_de_bienvenida

For the Spanish translation, I just added an entry on TranslateWiki for the help text (https://translatewiki.net/wiki/MediaWiki:Welcomesurvey-question-mailinglist-help/es), but I need the translation for the main label text in order to add it to (https://translatewiki.net/wiki/MediaWiki:Welcomesurvey-question-mailinglist-label/es).

@kostajh -- thanks; I just tried out the patchdemo a few different ways. The one thing that I can see that would be an improvement is to only show the checkbox if the user has entered an email address, either on Special:CreateAccount or in the field on Special:WelcomeSurvey. Do you see a simple way to do that? The rule could be as simple as "Appear if the user entered an email address on Special:CreateAccount OR has typed anything in the email box on Special:WelcomeSurvey".

If there is not a simple way to do it, then I don't think the consequences would be great if we skipped this nuance.

I thought about that, and it is fairly straightforward to do, but a couple of caveats:

  • it's additional complexity for a feature only shown temporarily to one wiki
  • if we go with the show/hide approach for the entire field, there will be edge cases for mobile where showing/hiding is a jarring experience for the user
  • if we disable/enable the checkbox, that is potentially a less surprising option for the end user, but we would probably want to add additional text explaining why the field is grayed out (more complexity)

So, given the above, it seems to me like it is OK to just leave it as is.

But if you prefer that we go with the show/hide or enable/disable approach, please let me know, and we'll do one of those options.

@kostajh -- thank you for explaining the alternatives. I agree that we should just leave it the way it is and show the checkbox for all users.

For the Spanish translation, I just added an entry on TranslateWiki for the help text (https://translatewiki.net/wiki/MediaWiki:Welcomesurvey-question-mailinglist-help/es), but I need the translation for the main label text in order to add it to (https://translatewiki.net/wiki/MediaWiki:Welcomesurvey-question-mailinglist-label/es).

It looks like a volunteer already did the translation https://translatewiki.net/wiki/MediaWiki:Welcomesurvey-question-mailinglist-label/es

So @MMiller_WMF this is good to be enabled on eswiki whenever you decide you would like that to happen. But it sounds like we don't want to start showing it until we have the export mechanism in place, right?

IMO we should make a separate task for the export tooling (could be a subtask of this one) since it has its own requirements, business logic etc that's separate from the display of the checkbox.

@MMiller_WMF - for your review some minor points:

  1. It's possible now to check the checkbox even if no email address was entered.
  1. Special:WelcomeSurvey won't display the email field if a user entered an email address on Special:CreateAccount. It doesn't seem to me to have any complications, although there might be couple of points for a user to get surprised. E.g. users might have chosen a different email if they had known that the email address can be used for receiving the editing tips.

@MMiller_WMF - for your review some minor points:

  1. It's possible now to check the checkbox even if no email address was entered.

On this point, could a low effort thing be to show a warning message if the checkbox is selected with no email entered saying something like Please enter a valid email address above in order to receive editing tips

  1. Special:WelcomeSurvey won't display the email field if a user entered an email address on Special:CreateAccount. It doesn't seem to me to have any complications, although there might be couple of points for a user to get surprised. E.g. users might have chosen a different email if they had known that the email address can be used for receiving the editing tips.

@MMiller_WMF - for your review some minor points:

  1. It's possible now to check the checkbox even if no email address was entered.

On this point, could a low effort thing be to show a warning message if the checkbox is selected with no email entered saying something like Please enter a valid email address above in order to receive editing tips

  1. Special:WelcomeSurvey won't display the email field if a user entered an email address on Special:CreateAccount. It doesn't seem to me to have any complications, although there might be couple of points for a user to get surprised. E.g. users might have chosen a different email if they had known that the email address can be used for receiving the editing tips.

I left some comments on this in T303240#7784768. The main thing is to reduce complexity and effort as we have lots of other things in progress right now (e.g. the warning message would need to toggle its visibility depending on whether the text in the input field is a valid email, we would need another translatable string, etc).

Also the user might (unlikely, of course) fill out their email later in Special:Preferences, or after being prompted on Special:Homepage.

IMO we should make a separate task for the export tooling (could be a subtask of this one) since it has its own requirements, business logic etc that's separate from the display of the checkbox.

Done with T304422: Welcome emails: Maintenance script for periodic exports

Change 773203 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):

[mediawiki/extensions/GrowthExperiments@master] WelcomeSurvey: Use experiment groups for showing/hiding mailing list question

https://gerrit.wikimedia.org/r/773203

Change 773204 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):

[operations/mediawiki-config@master] GrowthExperiments: Add mailing list question for eswiki

https://gerrit.wikimedia.org/r/773204

Moving back to code review, as I'm changing the mechanism for how we're displaying the checkbox.

@MMiller_WMF - for your review some minor points:

  1. It's possible now to check the checkbox even if no email address was entered.
  1. Special:WelcomeSurvey won't display the email field if a user entered an email address on Special:CreateAccount. It doesn't seem to me to have any complications, although there might be couple of points for a user to get surprised. E.g. users might have chosen a different email if they had known that the email address can be used for receiving the editing tips.

I am good with both of these points, and I agree with @kostajh that we should skip the warning text for the checkbox, just to keep things simple.

Change 775951 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):

[operations/mediawiki-config@master] GrowthExperiments: Start mailing list campaign on eswiki

https://gerrit.wikimedia.org/r/775951

Change 773203 merged by jenkins-bot:

[mediawiki/extensions/GrowthExperiments@master] WelcomeSurvey: Use experiment groups for showing/hiding mailing list question

https://gerrit.wikimedia.org/r/773203

Change 773204 merged by jenkins-bot:

[operations/mediawiki-config@master] GrowthExperiments: Add mailing list question for eswiki

https://gerrit.wikimedia.org/r/773204

Change 777397 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):

[mediawiki/extensions/GrowthExperiments@wmf/1.39.0-wmf.5] WelcomeSurvey: Use experiment groups for showing/hiding mailing list question

https://gerrit.wikimedia.org/r/777397

Change 777397 abandoned by Kosta Harlan:

[mediawiki/extensions/GrowthExperiments@wmf/1.39.0-wmf.5] WelcomeSurvey: Use experiment groups for showing/hiding mailing list question

Reason:

Not enough time in this window.

https://gerrit.wikimedia.org/r/777397

Mentioned in SAL (#wikimedia-operations) [2022-04-06T07:55:58Z] <kharlan@deploy1002> Synchronized wmf-config: Config: [[gerrit:773204|GrowthExperiments: Add mailing list question for eswiki (T303240 T305015)]] (duration: 00m 56s)

@Tgr - questions:

@Etonkovidova thank you for your diligence! I believe that link should point to their general terms of service, which is actually located at https://mailchimp.com/legal/terms/, but I've reached out to the Legal team to confirm as there is also a Spanish language version. I'll bring the answer back here as soon as I receive it.

Change 785165 had a related patch set uploaded (by Gergő Tisza; author: Gergő Tisza):

[operations/mediawiki-config@master] [beta] WelcomeSurveyExperimentalGroups: Use enwiki instead of eswiki

https://gerrit.wikimedia.org/r/785165

Change 785165 merged by jenkins-bot:

[operations/mediawiki-config@master] [beta] WelcomeSurveyExperimentalGroups: Use enwiki instead of eswiki

https://gerrit.wikimedia.org/r/785165

Right, sorry, that was also configured for beta eswiki (which is closed down). Should work with beta enwiki now.

@Etonkovidova thank you for your diligence! I believe that link should point to their general terms of service, which is actually located at https://mailchimp.com/legal/terms/, but I've reached out to the Legal team to confirm as there is also a Spanish language version. I'll bring the answer back here as soon as I receive it.

Thx, @EdErhart-WMF - it seems that the link https://mailchimp.com/es/legal/terms/ refers to the info that it's needed, but it's good to double-check.

@Etonkovidova that Spanish-language link is what we're looking for! I've updated the task description.

Change 785801 had a related patch set uploaded (by Gergő Tisza; author: Gergő Tisza):

[mediawiki/extensions/GrowthExperiments@master] Welcome emails: update opt-in checkbox copy

https://gerrit.wikimedia.org/r/785801

Change 785801 merged by jenkins-bot:

[mediawiki/extensions/GrowthExperiments@master] Welcome emails: update opt-in checkbox copy

https://gerrit.wikimedia.org/r/785801

Re-checked in eswiki betalabs - the links Política de privacidad and Condiciones de servicio are updated (per the task description and work as expected).

Note: Both links do not have target="_blank" attribute (clicking on the link will open the link in the same tab). Two other links on Special:WelcomeSurvey page (in the Getting started with editing section open in a different tab). This behavior is consistent with how external links behave on a wiki, but in the context of WelcomeSurvey it might be disruptive.

See T307844: Turn on GrowthExperiments welcome email campaign for actually enabling this in production. The issue pointed out by @Etonkovidova should probably be fixed first, though.

Change 789990 had a related patch set uploaded (by Gergő Tisza; author: Gergő Tisza):

[mediawiki/extensions/GrowthExperiments@master] Welcome emails: Open privacy footer links in new tab

https://gerrit.wikimedia.org/r/789990

The patch above fixes a technical issue with the landing page (see commit summary for details). In theory it might result in some small spacing changes, in practice it doesn't seem to:

beforeafter
desktop
dev.wiki.local.wmftest.net_22100_w_index.php_title=Special_CreateAccount&campaign=social-latam-2022-A&geNewLandingHtml=1&geEnabled=1 (1).png (948×1 px, 283 KB)
dev.wiki.local.wmftest.net_22100_w_index.php_title=Special_CreateAccount&campaign=social-latam-2022-A&geNewLandingHtml=1&geEnabled=1&mobileaction=toggle_view_desktop (1).png (948×1 px, 283 KB)
mobile
dev.wiki.local.wmftest.net_22100_w_index.php_title=Special_CreateAccount&campaign=social-latam-2022-A&geNewLandingHtml=1&geEnabled=1&mobileaction=toggle_view_mobile(Pixel 5) (1).png (2×1 px, 697 KB)
dev.wiki.local.wmftest.net_22100_w_index.php_title=Special_CreateAccount&campaign=social-latam-2022-A&geNewLandingHtml=1&geEnabled=1&mobileaction=toggle_view_mobile(Pixel 5) (2).png (2×1 px, 697 KB)

Change 789990 had a related patch set uploaded (by Gergő Tisza; author: Gergő Tisza):

[mediawiki/extensions/GrowthExperiments@master] Welcome emails: Open privacy footer links in new tab

https://gerrit.wikimedia.org/r/789990

I am not sure if this needs to be backported; moving to QA.

I am not sure if this needs to be backported; moving to QA.

It doesn't; we talked about it during the Monday meeting, it's a nice-to-have.
(Also, incorrectly titled - I guess I confused this task with the custom landing pages when I wrote the commit summary.)

Change 789990 merged by jenkins-bot:

[mediawiki/extensions/GrowthExperiments@master] Welcome emails: Open privacy footer links in new tab

https://gerrit.wikimedia.org/r/789990

Test wiki on Patch demo by KHarlan (WMF) using patch(es) linked to this task was deleted:

https://patchdemo.wmflabs.org/wikis/fa384c3ada/w/