Page MenuHomePhabricator

2017 Toolforge user survey
Closed, ResolvedPublic

Description

Run a survey to follow up to the 2016 Tool Labs user survey (T147336) in late October/early November 2017.

whoitemstatus
@leilaPrepare a first draft of the survey{{done}}
@bd808Check the draft survey{{done}}
@bd808Prepare a draft of the survey announce email{{done}}
@leilaRequest privacy statement{{done}} no change of substance to the survey from 2015 & 2016 version
@bd808Prepare a list of potential participants{{done}}
@bd808Finalize the date and update CE Calendar{{done}} scheduled for 2017-11-20 thru 2017-12-01
@bd808Send the survey announce email out{{done}}
@bd808Analysis

Changes to make from 2016 survey

  • Update terminology ("Tool Labs" -> "Toolforge")
  • Drop "service usage" questions

Event Timeline

bd808 triaged this task as Medium priority.Sep 29 2017, 11:40 PM

@leila Can we work on getting the survey form and its data collection backend setup soon? This week would be great if you have time.

A draft email announcement has been created at https://etherpad.wikimedia.org/p/ToolforgeAnnualSurveyEmail. It needs the survey link and the end date of the survey filled in before actually being used. The message is functionally the same as the 2016 email. A notable change is the opt-out mechanism. The new bulk mail sending script (T148783) supports reading a list of accounts to exclude from a wiki page. That page has been created at https://wikitech.wikimedia.org/wiki/Annual_Toolforge_Survey/Opt_out.

We have typically let the survey run for 2 weeks from the date of the initial email announcement.

This script can be used from silver.wikimedia.org to create a list of developer accounts having membership in the Toolforge project that also have validated email accounts and have not globally opt-ed out of email contact:

1#!/usr/bin/python
2# -*- coding: utf-8 -*-
3#
4# Generate a list of accounts for all Toolforge users who have
5# validated their email address via Wikitech and who have not opted out of
6# email contact.
7#
8# Copyright © 2017 Wikimedia Foundation and contributors
9# CC0 1.0 <https://creativecommons.org/publicdomain/zero/1.0/>
10from __future__ import print_function
11import ldap
12import subprocess
13
14
15l = ldap.open('ldap-labs.eqiad.wikimedia.org')
16l.protocol_version = ldap.VERSION3
17
18baseDN = 'dc=wikimedia,dc=org'
19searchScope = ldap.SCOPE_SUBTREE
20
21# Get a list of all Toolforge project members
22res = l.search_s(baseDN, searchScope, 'cn=project-tools', ['member'])
23members = res[0][1]['member']
24
25# Get the developer account name for all Toolforge project members
26ldap_names = []
27for member in members:
28 res = l.search_s(baseDN, searchScope, member.split(',')[0], ['cn'])
29 if res and 'cn' in res[0][1]:
30 ldap_names.append(res[0][1]['cn'][0])
31ldap_names = set(ldap_names)
32
33# Get the username for all Wikitech accounts that have validated their address
34# at some point and who have not opt-ed out of email contact by other users.
35query = """\
36SELECT user_name
37FROM user
38WHERE user_email <> ''
39 AND user_email_authenticated IS NOT NULL
40 AND user_id NOT IN (
41 SELECT up_user
42 FROM user_properties
43 WHERE up_property='disablemail'
44 AND up_value=1
45);"""
46sql = subprocess.Popen(
47 ['/usr/local/bin/sql', 'labswiki'],
48 stdout=subprocess.PIPE, stdin=subprocess.PIPE, stderr=subprocess.PIPE)
49mysql_names = set(sql.communicate(input=query)[0].strip().split('\n')[1:])
50
51# Find the intersection of the two lists
52users = [u for u in ldap_names if u in mysql_names]
53users.sort()
54for user in users:
55 print(user)

@leila I found an email thread between us and @egalvezwmf from last year's survey about adding an explicit privacy policy notice similar to https://wikimediafoundation.org/wiki/Wikipedia_Portal_Survey_Privacy_Statement to the survey. Is this something that I should ping someone in WMF-Legal about or is there a canned version somewhere that we can use? If I'm remembering correctly we have had a explanation of the data collected and its usage on the form itself in the past, but not a wiki page that contains the same information.

@bd808 yes! You should reach out to @APalmer_WMF. She can get you set up. It's a painless process.

@bd808

  • Re privacy statement, I'm reviewing the survey now and I'll follow this page which will give you a privacy statement. Please review the intro paragraphs to last year's survey and confirm that from your pov, this is what we want to do this year, too. (I went over them now and they still make sense to me.)
  • Has anyone opted out of the survey last year? They would email bd808 at wikimedia per instructions. If yes, please make sure you exclude them from the list of recipients for this year's survey.
  • If you have a way to link emails to wikitech usernames and the user has disabled "Email this user" feature, please interpret that as the user does not want to receive an email about the survey either.

I'm working on the survey itself now. will ping here once done.

leila updated the task description. (Show Details)

@bd808

  • Re privacy statement, I'm reviewing the survey now and I'll follow this page which will give you a privacy statement. Please review the intro paragraphs to last year's survey and confirm that from your pov, this is what we want to do this year, too. (I went over them now and they still make sense to me.)

The past information on privacy looks good to me. We do not explicitly collect username or IP address information in the backend google sheet that the survey form populates. We only share the results publicly in aggregate or summarized form. Only Foundation staff and NDA signed volunteers have access to the raw data and comments.

  • Has anyone opted out of the survey last year? They would email bd808 at wikimedia per instructions. If yes, please make sure you exclude them from the list of recipients for this year's survey.

Yes, we had some opt-outs last year. These are now captured on https://wikitech.wikimedia.org/wiki/Annual_Toolforge_Survey/Opt_out which will be honored by the new email sending script (T148783: Make maintenance script for sending annual survey emails). The opt-out process for the survey going forward will be for users to add their account name to that page. This is reflected in the new draft email that I have created.

  • If you have a way to link emails to wikitech usernames and the user has disabled "Email this user" feature, please interpret that as the user does not want to receive an email about the survey either.

This is handled by the script I posted as P6305 that will be used to create the mailing list.

@bd808 looks good to me. I'm done for now, unless you ping me or we hear about the privacy statement.

I'm going to put the dates for the survey into the CE calendar as 2017-11-20 through 2017-12-01. I think that should work assuming that we get the privacy statement done this week. It looks like everything else is basically ready to go.

Mentioned in SAL (#wikimedia-operations) [2017-11-20T17:19:57Z] <bd808> Sending Toolforge survey emails from silver for T177126

Initial email send complete:

$ ./send.sh | tee 20171120.log
...
[2017-11-20 19:09:27] processed: 1289 (0.2/sec); ok: 1281 (99.4%); failed: 0 (0.0%); missing: 0 (0.0%); norecieve: 0 (0.0%); optedout: 8 (0.6%);
done.

Mentioned in SAL (#wikimedia-operations) [2017-11-27T18:05:17Z] <bd808> Sending Toolforge survey 1 week reminder emails from silver for T177126

Mentioned in SAL (#wikimedia-operations) [2017-11-30T20:40:40Z] <bd808> Sending Toolforge survey final reminder emails from silver for T177126

I haven't done the analysis yet.

@bd808 apologies I closed this by accident when bulk-archiving our tasks marked as completed. Is @leila still working on this or should this be assigned to you?

@bd808 apologies I closed this by accident when bulk-archiving our tasks marked as completed. Is @leila still working on this or should this be assigned to you?

No worries. It's all in my court. I just need to write a blog post & email on the analysis that I have made on meta to promote the findings with our community. This is something I should have finished months ago.