@Urbanecm reports that he received a new-style verification email, the kind we developed for T215665: [Design] Emailability: Improve verification email. But we decided against enabling that feature months ago, and as far as we knew it was never enabled anywhere.
Description
Description
Details
Details
Related Changes in Gerrit:
| Subject | Repo | Branch | Lines +/- | |
|---|---|---|---|---|
| Set wgGEConfirmEmailEnabled to false for all wikis | operations/mediawiki-config | master | +3 -0 |
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | None | T233363 New verification emails unexpectedly enabled on cswiki, kowiki, arwiki, viwiki | |||
| Resolved | • nettrom_WMF | T233749 Investigate changes in email verification around June 1, 2019 |
Event Timeline
Comment Actions
It turns out this is a regression from rEGRE5ef7faffb42e: Invert configuration, defaulting features to "on", for T229389: Invert configuration for GrowthExperiments.
Comment Actions
Change 538100 had a related patch set uploaded (by Catrope; owner: Urbanecm):
[operations/mediawiki-config@master] Set wgGEConfirmEmailEnabled to false for all wikis
Comment Actions
This is on hold while @MMiller_WMF decides whether we should disable it, or keep it enabled. It's been enabled for about two and a half weeks now, since September 3rd at 12:19 UTC (which is when wmf.20 was finally fully rolled out; this was supposed to happen on August 29th, but was delayed).
Comment Actions
Thanks for reporting this, @Urbanecm. Although we didn't intend to release this feature, now that it's out, I think it's not urgent to pull it back because my brief analysis doesn't make it seem like it's impacting users negatively. I think we can use this opportunity to look at whether the updated email seems to be having any effect.
@nettrom_WMF -- I looked today as best I could with the user tables, and it doesn't look to me like the rate at which new accounts are confirming email addresses has gone up or down since Sep 3, but I would like if you could take a quick look, because you have better queries (like to take out autocreated accounts, etc). Would it be possible to do that in the next couple days, so we could decide whether to leave this feature on or pull it back, by, say Tuesday?
Comment Actions
@RHo -- what do you make of these results? How surprised are you to not see any change in the email verification rate? Does this make you want to change the feature, leave it the way it is, or remove it?
Comment Actions
It's a little disappointing, but at the same time glad to see there was no significant negative effect! And since it is a clearer confirmation email design, I think it could be left the way it is for now (seems wrong to revert back to the old message now that the work cleaning it up is done). We can maybe explore alternate designs to see if it makes a difference if and when we revisit the goal of increasing confirmed emails down the track.
Having said that, I have some questions (bargaining?) for @nettrom_WMF:
- Since this went out on our test wikis that have homepage, is it possible the email verification rate was already "lifted" through prompts onwiki?
- Is it worthwhile running the query after a little more time has passed?
- if so, is it possible to see if split by desktop and mobile makes any difference?
Comment Actions
As done in the previous comments (which I've deleted due to them using erroneous data), I decided to reuse the code and graphs we had for our analysis around emails (T204785), where we got proportion of registrations. In order to reflect any changes around deployment of the Homepage to Czech and Korean Wikipedias in May, the data gathering starts on 2019-01-01. Auto-created accounts are excluded from the analysis.
From what I can tell, there has not been any significant change in proportion of users who verify their email address after September 3 (in the graphs below, that's where the vertical dotted line is). The dip in Arabic seems unrelated and correlated with the dip in registrations w/email addresses). The red line is a 7-day moving average to make the trends easier to spot.
It's also not clear if the Homepage has had a significant effect on verification by looking at these graphs, but that can be difficult to ascertain because the graphs don't separate the treatment and control groups in that experiment.
Czech:
Korean:
Vietnamese:
Arabic:
Comment Actions
Thanks, @nettrom_WMF. Seeing these graphs, I think we should leave the changed emails on for a few more weeks. Then we can re-run these graphs and take another look. The emails seem not to have harmed anything, and I'd like to know a little more before deciding what to do.
I'm putting this in our Upcoming Work column to remind us to revisit.
Comment Actions
Change 538100 abandoned by Jforrester:
Set wgGEConfirmEmailEnabled to false for all wikis
Reason:
We're intentionally not disabling this, per Marshall.