Page MenuHomePhabricator

New verification emails unexpectedly enabled on cswiki, kowiki, arwiki, viwiki
Open, LowPublic

Description

@Urbanecm reports that he received a new-style verification email, the kind we developed for T215665: Emailability: Improve verification email. But we decided against enabling that feature months ago, and as far as we knew it was never enabled anywhere.

Details

Related Gerrit Patches:
operations/mediawiki-config : masterSet wgGEConfirmEmailEnabled to false for all wikis

Event Timeline

Catrope created this task.Sep 19 2019, 8:40 PM
Restricted Application added a project: Growth-Team. · View Herald TranscriptSep 19 2019, 8:40 PM
Restricted Application added subscribers: alanajjar, revi, Aklapper. · View Herald Transcript

Change 538100 had a related patch set uploaded (by Catrope; owner: Urbanecm):
[operations/mediawiki-config@master] Set wgGEConfirmEmailEnabled to false for all wikis

https://gerrit.wikimedia.org/r/538100

Catrope renamed this task from New verification emails unexpectedly enabled on cswiki. kowiki, arwiki, viwiki to New verification emails unexpectedly enabled on cswiki, kowiki, arwiki, viwiki.Sep 19 2019, 8:46 PM

This is on hold while @MMiller_WMF decides whether we should disable it, or keep it enabled. It's been enabled for about two and a half weeks now, since September 3rd at 12:19 UTC (which is when wmf.20 was finally fully rolled out; this was supposed to happen on August 29th, but was delayed).

MMiller_WMF added a subscriber: nettrom_WMF.

Thanks for reporting this, @Urbanecm. Although we didn't intend to release this feature, now that it's out, I think it's not urgent to pull it back because my brief analysis doesn't make it seem like it's impacting users negatively. I think we can use this opportunity to look at whether the updated email seems to be having any effect.

@nettrom_WMF -- I looked today as best I could with the user tables, and it doesn't look to me like the rate at which new accounts are confirming email addresses has gone up or down since Sep 3, but I would like if you could take a quick look, because you have better queries (like to take out autocreated accounts, etc). Would it be possible to do that in the next couple days, so we could decide whether to leave this feature on or pull it back, by, say Tuesday?

[comment deleted due to graphs using erroneous data, see updated comment below]

nettrom_WMF moved this task from Triage to Review on the Product-Analytics board.Sep 20 2019, 8:46 PM

@RHo -- what do you make of these results? How surprised are you to not see any change in the email verification rate? Does this make you want to change the feature, leave it the way it is, or remove it?

RHo added a comment.Sep 23 2019, 11:10 PM

@RHo -- what do you make of these results? How surprised are you to not see any change in the email verification rate? Does this make you want to change the feature, leave it the way it is, or remove it?

It's a little disappointing, but at the same time glad to see there was no significant negative effect! And since it is a clearer confirmation email design, I think it could be left the way it is for now (seems wrong to revert back to the old message now that the work cleaning it up is done). We can maybe explore alternate designs to see if it makes a difference if and when we revisit the goal of increasing confirmed emails down the track.

Having said that, I have some questions (bargaining?) for @nettrom_WMF:

  • Since this went out on our test wikis that have homepage, is it possible the email verification rate was already "lifted" through prompts onwiki?
  • Is it worthwhile running the query after a little more time has passed?
    • if so, is it possible to see if split by desktop and mobile makes any difference?
nettrom_WMF added a comment.EditedSep 24 2019, 12:46 AM

[graphs removed as they were based on erroneous data]

As done in the previous comments (which I've deleted due to them using erroneous data), I decided to reuse the code and graphs we had for our analysis around emails (T204785), where we got proportion of registrations. In order to reflect any changes around deployment of the Homepage to Czech and Korean Wikipedias in May, the data gathering starts on 2019-01-01. Auto-created accounts are excluded from the analysis.

From what I can tell, there has not been any significant change in proportion of users who verify their email address after September 3 (in the graphs below, that's where the vertical dotted line is). The dip in Arabic seems unrelated and correlated with the dip in registrations w/email addresses). The red line is a 7-day moving average to make the trends easier to spot.

It's also not clear if the Homepage has had a significant effect on verification by looking at these graphs, but that can be difficult to ascertain because the graphs don't separate the treatment and control groups in that experiment.

Czech:

Korean:

Vietnamese:

Arabic:

Thanks, @nettrom_WMF. Seeing these graphs, I think we should leave the changed emails on for a few more weeks. Then we can re-run these graphs and take another look. The emails seem not to have harmed anything, and I'd like to know a little more before deciding what to do.

I'm putting this in our Upcoming Work column to remind us to revisit.

MMiller_WMF moved this task from Inbox to Upcoming Work on the Growth-Team board.Sep 24 2019, 9:40 PM

Change 538100 abandoned by Jforrester:
Set wgGEConfirmEmailEnabled to false for all wikis

Reason:
We're intentionally not disabling this, per Marshall.

https://gerrit.wikimedia.org/r/538100

nettrom_WMF triaged this task as Low priority.Nov 1 2019, 6:16 PM
nettrom_WMF moved this task from Review to Tracking on the Product-Analytics board.