|operations/mediawiki-config : master||Set wgGEConfirmEmailEnabled to false for all wikis|
|Open||None||T233363 New verification emails unexpectedly enabled on cswiki, kowiki, arwiki, viwiki|
|Resolved||nettrom_WMF||T233749 Investigate changes in email verification around June 1, 2019|
- Mentioned In
- T233749: Investigate changes in email verification around June 1, 2019
- Mentioned Here
- T204785: Growth: how many users sign up with email addresses?
T229389: Invert configuration for GrowthExperiments
rEGRE5ef7faffb42e: Invert configuration, defaulting features to "on"
T215665: Emailability: Improve verification email
This is on hold while @MMiller_WMF decides whether we should disable it, or keep it enabled. It's been enabled for about two and a half weeks now, since September 3rd at 12:19 UTC (which is when wmf.20 was finally fully rolled out; this was supposed to happen on August 29th, but was delayed).
Thanks for reporting this, @Urbanecm. Although we didn't intend to release this feature, now that it's out, I think it's not urgent to pull it back because my brief analysis doesn't make it seem like it's impacting users negatively. I think we can use this opportunity to look at whether the updated email seems to be having any effect.
@nettrom_WMF -- I looked today as best I could with the user tables, and it doesn't look to me like the rate at which new accounts are confirming email addresses has gone up or down since Sep 3, but I would like if you could take a quick look, because you have better queries (like to take out autocreated accounts, etc). Would it be possible to do that in the next couple days, so we could decide whether to leave this feature on or pull it back, by, say Tuesday?
It's a little disappointing, but at the same time glad to see there was no significant negative effect! And since it is a clearer confirmation email design, I think it could be left the way it is for now (seems wrong to revert back to the old message now that the work cleaning it up is done). We can maybe explore alternate designs to see if it makes a difference if and when we revisit the goal of increasing confirmed emails down the track.
Having said that, I have some questions (bargaining?) for @nettrom_WMF:
- Since this went out on our test wikis that have homepage, is it possible the email verification rate was already "lifted" through prompts onwiki?
- Is it worthwhile running the query after a little more time has passed?
- if so, is it possible to see if split by desktop and mobile makes any difference?
As done in the previous comments (which I've deleted due to them using erroneous data), I decided to reuse the code and graphs we had for our analysis around emails (T204785), where we got proportion of registrations. In order to reflect any changes around deployment of the Homepage to Czech and Korean Wikipedias in May, the data gathering starts on 2019-01-01. Auto-created accounts are excluded from the analysis.
From what I can tell, there has not been any significant change in proportion of users who verify their email address after September 3 (in the graphs below, that's where the vertical dotted line is). The dip in Arabic seems unrelated and correlated with the dip in registrations w/email addresses). The red line is a 7-day moving average to make the trends easier to spot.
It's also not clear if the Homepage has had a significant effect on verification by looking at these graphs, but that can be difficult to ascertain because the graphs don't separate the treatment and control groups in that experiment.
Thanks, @nettrom_WMF. Seeing these graphs, I think we should leave the changed emails on for a few more weeks. Then we can re-run these graphs and take another look. The emails seem not to have harmed anything, and I'd like to know a little more before deciding what to do.
I'm putting this in our Upcoming Work column to remind us to revisit.