Page MenuHomePhabricator

Test the model for eliciting new editor interests
Closed, ResolvedPublic

Description

https://www.mediawiki.org/wiki/Wikimedia_Technology/Goals/2017-18_Q4#Program_12:_Grow_contributor_diversity

Description of the experiment
https://meta.wikimedia.org/wiki/Research:Voice_and_exit_in_a_voluntary_work_environment/Elicit_new_editor_interests#Online_Experiment

Audience of the experiment
Those who register in Wikipedia in the period of the experiment.

Project
English Wikipedia

Duration
One week first. If enough data is collected, stop at the end of the first week. If not, continue for a second week.

Event Timeline

bmansurov updated the task description. (Show Details)
leila claimed this task.Mar 29 2018, 7:42 PM
leila triaged this task as High priority.
leila moved this task from Staged to Time Sensitive on the Research board.Mar 29 2018, 9:46 PM
leila moved this task from Time Sensitive to In Progress on the Research board.Apr 5 2018, 10:46 PM
leila added a comment.Apr 13 2018, 8:42 PM

Update: Bob, Ramtin, and I talked about the details of how the experiment can look like, and we need around a week or two to sort out details. I'm off on the week on May 1st, so the experiment itself can realistically start in 3 weeks. In the mean time, of course, there is work to do:

  • Choice of language: it makes most sense to run this test in English. Running it in another language right now requires updating the code base to handle text in other languages and this needs some days of coding and quality check. We should not do this for this very first experiment. So, let's fix the language for this to en.
  • We will be doing user testing and that's in the realm of Abbey and Jonathan's work. I'll reach out to Jonathan to see if he has time to talk with Bob, Ramtin and I about details next week.
leila updated the task description. (Show Details)Apr 13 2018, 8:46 PM
leila updated the task description. (Show Details)Apr 13 2018, 8:48 PM
leila updated the task description. (Show Details)
leila updated the task description. (Show Details)May 25 2018, 11:09 PM

@MMiller_WMF if you're interested, please review the task description, especially the section on Description of the experiment. The details are sorted out now and we are close to the experiment point. I'll wait until 2018-05-31 for your feedback and then set the start time of the experiment accordingly. Can you also let us know if you can support us with sending emails to the recipients?

@bmansurov can you look into how we can send emails to enwiki users who have registered a Wikimedia account in a specific time period, excluding users that have opted out from being contacted by other users? Just FYI, the last time we send these emails was from Wikimedia Research [mailto:recommender-feedback@wikimedia.org]. For context, we want to send emails out every n hours, and as users register in a period of a week on enwiki. @MMiller_WMF feel free to jump in if you already have a solution for this, of course. :)

@leila, I've looked into sending emails to recently registered users. We can get the list by running an SQL query:

echo "select user_name, user_registration from enwiki.user order by user_id desc limit 60000;" | mysql -h analytics-store.eqiad.wmnet -A > users.txt

Note that filtering by user_registration is unbearably slow because the field is not indexed. We can post-process data to only get the users within a certain time frame. We can dry run SendBulkEmails.php to get the number of users who don't want to receive emails or blocked recommender-feedback@wikimedia.org. Depending on the number we can feed more/fewer users to the script.

leila added a comment.EditedMay 29 2018, 5:23 PM

@leila, I've looked into sending emails to recently registered users. We can get the list by running an SQL query:

echo "select user_name, user_registration from enwiki.user order by user_id desc limit 60000;" | mysql -h analytics-store.eqiad.wmnet -A > users.txt

Note that filtering by user_registration is unbearably slow because the field is not indexed. We can post-process data to only get the users within a certain time frame.

Ok. So in practice we should be able to do this, for example, twice a day and send emails out, right?

We can dry run SendBulkEmails.php to get the number of users who don't want to receive emails or blocked recommender-feedback@wikimedia.org. Depending on the number we can feed more/fewer users to the script.

Let's give it a try and do a dry-run, maybe with our own volunteer accounts.

Ok. So in practice we should be able to do this, for example, twice a day and send emails out, right?

Yes.

Let's give it a try and do a dry-run, maybe with our own volunteer accounts.

OK!

Let's give it a try and do a dry-run, maybe with our own volunteer accounts.

Count me in. :)

FYI, I'm waiting to get access to send emails (although I have access to log into the server, I don't have access to send emails yet): T189285#4240614.

I've got the access and here are the results of the dry-run (i.e. you shouldn't have received any mails):

WARNING - User Bmansurov_(WMF) can't receive mail
INFO - Emailing LZia_(WMF) <leila@wikimedia.org>
INFO - Emailing Trizek_(WMF) <bevellin@wikimedia.org>
[2018-06-01 14:28:55] processed: 3 (0.3/sec); ok: 2 (66.7%); failed: 0 (0.0%); missing: 0 (0.0%); noreceive: 1 (33.3%); optedout: 0 (0.0%);
done.

This seems correct to me because I've opted out of receiving emails. I suppose you two didn't.

@leila, do we have access to the recommender-feedback@wikimedia.org account? We need to create a wiki account with this email address if we want it to be the sender. We can then create a subpage under this user's account that lists the usernames that opt out of emails. Once we create this account, I'll send test emails to us all.

leila added a subscriber: ori.Jun 4 2018, 11:14 PM

I've got the access and here are the results of the dry-run (i.e. you shouldn't have received any mails):

WARNING - User Bmansurov_(WMF) can't receive mail
INFO - Emailing LZia_(WMF) <leila@wikimedia.org>
INFO - Emailing Trizek_(WMF) <bevellin@wikimedia.org>
[2018-06-01 14:28:55] processed: 3 (0.3/sec); ok: 2 (66.7%); failed: 0 (0.0%); missing: 0 (0.0%); noreceive: 1 (33.3%); optedout: 0 (0.0%);
done.

This seems correct to me because I've opted out of receiving emails. I suppose you two didn't.

yup. looks right.

@leila, do we have access to the recommender-feedback@wikimedia.org account?

@ori set that account. Not sure if he still has access to it.

@Aklapper o/ @ori is no longer with us. Would you be able to make one of us an admin of the recommender-feedback@wikimedia.org?

leila added a comment.Jun 5 2018, 9:33 PM

@Trizek-WMF is it possible that we send the announcement to enwiki while bmansurov is finalizing the email component? We know that we can start no later than 2 weeks from now, so we might as well notify the community to save some time with parallelization. What do you think?

You can send the announcement on en.wp anytime, if you are ready to potentially spend time there to reply to questions.

Focus on the fact that it will not impact active users and what are the future benefits for active users.

leila added a comment.Jun 6 2018, 6:24 PM

You can send the announcement on en.wp anytime, if you are ready to potentially spend time there to reply to questions.

I'm ready. Which section is the most appropriate?

Focus on the fact that it will not impact active users and what are the future benefits for active users.

yup.

leila added a comment.Jun 6 2018, 6:37 PM

On it. Thanks!

@leila, while we're waiting on T196556, I've sent test emails to @leila and @Trizek-WMF. Let me know if we should proceed with using my email as the sender and be done with the emails.

leila added a comment.Jun 8 2018, 5:12 PM

@bmansurov I confirm that I've received your email.

It's better to wait for T196556 as we need a more formal looking account, ideally. Let's wait for that.

leila added a comment.Jun 12 2018, 1:31 AM

prepared the form for privacy statement request. The form is with Ramtin for his review.

@leila, I've sent a test email. Unsubscribe workflow also seems to work. Waiting on the subject of the email and a couple of other things we talked about today.

leila added a comment.Aug 6 2018, 11:07 PM

The experiment started today.

Izno added a subscriber: Izno.Aug 7 2018, 12:13 AM
bmansurov added a comment.EditedAug 8 2018, 8:59 PM

@leila, Here's the histogram of the minutes to first edit after registration from 07/05/2018 till 08/05/2018. During this time 43,436 users made their first edit. To briefly summarize, about 41,000 users made their first edit within the first 3 days. I'm also attaching the raw data below.

Context: https://meta.wikimedia.org/wiki/Research_talk:Voice_and_exit_in_a_voluntary_work_environment/Elicit_new_editor_interests

leila added a comment.Aug 9 2018, 4:32 PM

@Trizek-WMF just responded on that task. sorry about the delay. I somehow missed the ping there.

leila added a comment.Aug 21 2018, 2:45 PM

I sent an update to EN-VP that we are done with the first stage of the experiment and starting the second stage: https://en.wikipedia.org/wiki/Wikipedia:Village_pump_(miscellaneous)#Experiment:_Eliciting_New_Editor_Interests_[Update]

We have a new experiment based on the learnings from the previous one. I put a note on enwiki's VP about it https://en.wikipedia.org/wiki/Wikipedia:Village_pump_(miscellaneous)#Experiment:_Eliciting_New_Editor_Interests_(Update)

leila added a comment.Dec 13 2018, 5:38 PM

VP and the research talk page are quiet after the first batch of emails that went out yesterday. Let's proceed with the second batch after coordinating on IRC about a few questions. Stats of this round are being updated daily at https://meta.wikimedia.org/wiki/Research:Voice_and_exit_in_a_voluntary_work_environment/Elicit_new_editor_interests#Stats

leila closed this task as Resolved.Jan 15 2019, 8:17 PM