Page MenuHomePhabricator

[REQUEST] Estimate the number of "emailable" editors
Closed, ResolvedPublic

Description

What's requested: At any given point in time, how many active users have (1) provided an email address and (2) indicated that it's okay to "email this user"? An estimate is completely fine, I would assume this is a moving target over time.

Why it's requested: L&E is working with Legal to revise our dissemination practices for the Community Insights survey, and it would be helpful to know the % of active users we'd be able to sample from if we targeted only those who indicated that it was okay to email them.

When it's requested: The end of April, if possible.

Other helpful information:

Event Timeline

Rmaung created this task.Apr 15 2020, 3:24 PM
nshahquinn-wmf renamed this task from [REQUEST] to [REQUEST] Estimate the number of "emailable" editors.Apr 16 2020, 11:47 AM

@Rmaung, thank you for filing this! We'll prioritize this at our upcoming triage meeting on Monday, 20 April.

In the meantime, one question: when you talk about indicating that it's okay to "email this user", are you referring to the preference below?

@nshahquinn-wmf, Thank you! As I only have a WMF account and don't see that option, I'm following up with Legal now to make sure this was the option we had discussed. I'll let you know here as soon as I hear back!

@nshahquinn-wmf, yes, that is the preference we're interested in!

LGoto assigned this task to nettrom_WMF.Apr 27 2020, 4:58 PM
LGoto edited projects, added Product-Analytics (Kanban); removed Product-Analytics.

I've completed a first pass of answering the two questions given in the task description. The way I've gone about doing this is to measure it on a per-wiki basis, and from that draw a reasonable estimate. I know that we can measure "active editors" across all wikis, since we report that monthly. I do suspect that users tend to be localized to one or a few wikis, they might disable email on one wiki and not others, and I'd prefer not to dig into global preferences in order to figure out if that's something people use yet. A side-effect of measuring this on a per-wiki basis is also that we can understand if there appears to be a relationship between size of the wiki/community and whether users turn email off.

I chose to start with the Wiki Comparison dataset. This was also the starting point in a previous analysis (T237129). As was done then, I used the top 10 wikis by size rank as a starting point. In order to understand how this plays out on other wikis, I selected three other groups of 10 wikis. Those were ranked 101–110, 201–210, and 301–310. The top 10 wikis are nine Wikipedias and Commons, while in the other groups we find lots of other wikis.

For each of the 40 wikis, I identified active editors for March using the same dataset as our monthly reports. Bots were excluded, and an "active editor" means an account that had 5 or more content edits during the month of March. For each editor, we identify whether they have an email address associated with their account, whether the email address is verified, and whether they have disabled other users from emailing them.

Q1: How many active editors have an email address?

This tends to be "at least 75%. One notable exception is among the top 10, and that's the Japanese Wikipedia with 60%. As size of the wiki goes down, we often see that all active editors have email addresses.

Among the top 10 wikis, the range is from 75.5% (English) to 87.4% (Russian).

Another way to answer this is that if you rely on emails to contact active editors, you're likely to miss 13–25% of them. If we use English Wikipedia as an example: it had more than 32,000 active editors in March, and that means around 8,000 don't have an email address.

Q2: How many have indicated that it's okay to email them?

For smaller wikis, this tends to be "all of them". For the top 10, we see that 2.9–6.3% of active editors have disabled email, meaning that 93.7–97.1% of them allow other users to email them. Commons is on the low end here with 2.9%, and German is the high end with 6.3%.

These proportions are cumulative, so if we again look at English Wikipedia and ballpark it: 25% don't have an email address, and another 5% have disabled it, so that gives you 30% (or around 9,500 users) can't be reached that way.

@Rmaung : I hope this is a meaningful estimate. Let me know if it isn't, i.e. I can measure it across all top 10 wikis and give two measurements if that's more useful. And please let me know what questions you might have about this!

kzimmerman closed this task as Resolved.Jun 2 2020, 5:20 PM
kzimmerman added a subscriber: kzimmerman.

Morten emailed @Rmaung to check in last week; given we haven't heard feedback I'm going to close this as resolved. @Rmaung please let us know if you have questions!