Page MenuHomePhabricator

2016 Tool Labs user survey
Closed, ResolvedPublic

Description

Run a survey to follow up to the 2015 Tool Labs user survey (T95155)

itemstatuswho
Prepare a first draft of the survey{{Done}} https://docs.google.com/a/wikimedia.org/forms/d/1OdzFNA14ob0CuYdB3fhDTllXuaAWuwHV53GIcyxauYM/edit#@leila
Check the draft with some staff and volunteers{{Done}} Note: we did this step last year. If we didn't have any feedback in the mean time, I suggest we skip this step. --Leila@bd808, @leila, @yuvipanda
Email draft{{Done}} Note: link to last year's email https://etherpad.wikimedia.org/p/ToolLabsAnnualSurvey@bd808
Run the survey by Legal{{Done}} Note: The survey is exactly as last year's. There is no need to run it by Legal again as long as we don't change the data retention and handling terms. --Leila@leila
Prepare a list of potential participants{{Done}} T147336#2723747@bd808,@yuvipanda
Finalize the date and update CE Calendar{{Done}} (diff)@bd808 , @leila
Send the survey out out{{Done}}@bd808
Analysis{{Done}} (published on meta)@bd808, @leila

Event Timeline

bd808 created this task.Oct 4 2016, 5:54 PM
Restricted Application added a project: Cloud-Services. · View Herald TranscriptOct 4 2016, 5:54 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
leila reassigned this task from leila to bd808.Oct 11 2016, 7:04 PM
leila updated the task description. (Show Details)
Restricted Application added a project: User-bd808. · View Herald TranscriptOct 11 2016, 7:04 PM
leila added a comment.Oct 11 2016, 7:05 PM

@bd808 Please review the survey linked from the description, if you are happy, let's set a date and run it. I'll share the email we sent out last year shortly.

leila updated the task description. (Show Details)Oct 11 2016, 7:42 PM
leila added a project: Research.
leila added a subscriber: Research.
leila removed a subscriber: Research.

@bd808 Please review the survey linked from the description, if you are happy, let's set a date and run it. I'll share the email we sent out last year shortly.

I changed the private contact email from @yuvipanda to me, but other than that I think it looks good. Keeping the survey questions and order the same for this second iteration of the survey should give us comparable results.

I think we can basically make the same changes to last year's email message. That probably means the only thing holding us up from sending the email is gathering the list of email addresses from Wikitech.

Thank you for taking it on, @bd808 / @leila. I've one person who wanted to
'opt out', I'll share that with you two.

leila added a comment.Oct 17 2016, 1:07 PM

@bd808 this sounds good. When do you want it to go out? :) 2016-10-20, for example?

leila updated the task description. (Show Details)Oct 17 2016, 1:09 PM

@bd808 this sounds good. When do you want it to go out? :) 2016-10-20, for example?

2016-10-20 sounds like a start time we can hit. It looks like we kept the survey open for 2 weeks last year so this iteration's close date would be something like 2016-11-04.

bd808 moved this task from To Do to In Dev/Progress on the User-bd808 board.Oct 17 2016, 3:20 PM
bd808 updated the task description. (Show Details)Oct 17 2016, 4:18 PM

The list of email addresses to send the survey to can be generated by running P4254 on silver.wikimedia.org.

bd808 updated the task description. (Show Details)Oct 17 2016, 8:17 PM

I have a mass mailing script ready to go for actually sending the email. It takes into account some feedback from last year that using Bcc: for the recipients may have caused some emails to go to spam. It will send individual messages with the recipient in the To: line.

bd808 moved this task from Triage to In Progress on the Toolforge board.Oct 17 2016, 10:45 PM
bd808 updated the task description. (Show Details)Oct 18 2016, 11:16 PM

I have stubbed out https://meta.wikimedia.org/wiki/Research:Annual_Tool_Labs_Survey/2016 as the location to report the survey results. Hopefully we can report some year over year comparisons on the main https://meta.wikimedia.org/wiki/Research:Annual_Tool_Labs_Survey page as well.

@bd808

  • Just to make sure: you will exclude the one user who had asked to be opted out from future emails, per Yuvi?
  • I read the 2016 email template. Looks good to me.
  • Re your signature, please follow last year's signature and add items to that list as you see fit. This is the recommendation signature under your name and title:

Wikimedia Foundation
149 New Montgomery Street, 6th Floor
San Francisco, CA 94105
415.839.6885 (Office)
415.882.0495 (Fax)

  • Just to make sure: you will exclude the one user who had asked to be opted out from future emails, per Yuvi?

Sadly, the one manual opt-out email address has been lost to the sands of time. Hopefully the user will be understanding. I have a place setup on a trusted server to record any new opt-outs that are received this time around.

Emails are going out!

Mentioned in SAL (#wikimedia-operations) [2016-10-20T22:52:02Z] <bd808> Finished sending Tool Labs survey emails from silver (T147336)

bd808 updated the task description. (Show Details)Oct 27 2016, 8:39 PM
bd808 triaged this task as Medium priority.Oct 28 2016, 3:17 PM
leila added a comment.Oct 28 2016, 7:36 PM

@bd808 feel free to send a reminder that the survey will close in a week. You can also do this 3 days before the survey closes.

Mentioned in SAL (#wikimedia-operations) [2016-10-28T20:42:29Z] <bd808> Sending Tool Labs survey reminder emails from silver (T147336)

Mentioned in SAL (#wikimedia-operations) [2016-11-02T18:11:21Z] <bd808> Sending final Tool Labs survey reminder emails from silver (T147336)

I have turned off the survey form. We received 175 raw responses.

One interesting thing I see from a very high level is that there were significantly higher response rates on the days that the 2 reminder emails went out. This looks to be a validation of the usefulness of sending reminders.

@bd808 I'm removing Research-and-Data, but I myself will stay on this task. Let's discuss the analysis whenever you're ready. :)

@leila Do you have time to do the initial pass at making a normalized set from the survey data like the spreadsheet that existed when I got involved in the last analysis? I'm not sure if that required a lot of work for you or if it was mostly what the google survey dumped out by default.

From there I can attempt to recreate the analysis that I did for the 2015 survey as a year-over-year comparison and then maybe dig in deeper to see if our responses show some other interesting data this time around.

leila added a comment.Nov 9 2016, 1:04 AM

@bd808 I used the data right from the survey, no normalization or anything fancy had been done. Feel free to apply the analysis you had from last year to this data. If anything surprising happens, let me know and I'll have a look at what may have gone wrong.

bd808 moved this task from Triage to In Progress on the Cloud-Services board.Nov 24 2016, 9:42 PM
bd808 closed this task as Resolved.Nov 28 2016, 3:39 AM
bd808 updated the task description. (Show Details)
bd808 moved this task from In Dev/Progress to Done on the User-bd808 board.Jan 7 2017, 12:19 AM
bd808 moved this task from Done to Archive on the User-bd808 board.