Page MenuHomePhabricator

Identify survey services compatible with our privacy policy
Closed, ResolvedPublic

Description

As WMF's research efforts expand (great!), we're seeing more and more surveys. Some of those are being linked from MediaWiki itself, see for instance Qualtrics at T1005 and previously SurveyMonkey for MediaViewer.

It can't be given for granted that sending users to third party websites from our interface is allowed by the privacy policy. It would be nice to determine what service are fine to link and document the outcome, for instance at the master list of software used by WMF, https://meta.wikimedia.org/wiki/FLOSS-Exchange

Note that this matter is distinct from the questions 1) whether it's appropriate to use unfree software, 2) what software is statistically suitable, and 3) how multilingual they are/how good their i18n and integration with the Translate extension. The discussion about that was at http://thread.gmane.org/gmane.org.wikimedia.analytics/345/

Services used by WMF

(Both are not usable by an EU entity due to lack of privacy policy and due to transfer of private data to USA soil.)

Not recommended but allowed

  • Google Docs is sometimes used for events feedback etc. and by the community because it's very easy to set up, sometimes widely advertised e.g. id.wikipedia sitenotice. Google's privacy policy is not consistent with Wikimedia's since they collect much more information from users. That being said, if we clearly indicate to users that the survey they are taking will be on Google and thus subject to the Google privacy policy (preferably before they click on the link to the survey), you can use it to administer the services.

Not evaluated by the Wikimedia Foundation yet

Event Timeline

Nemo_bis created this task.Apr 2 2015, 7:33 AM
Nemo_bis raised the priority of this task from to Normal.
Nemo_bis updated the task description. (Show Details)
Nemo_bis added subscribers: Nemo_bis, aripstra, MarkTraceur, leila.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 2 2015, 7:33 AM
Nemo_bis updated the task description. (Show Details)Apr 2 2015, 7:42 AM
Nemo_bis set Security to None.
Nemo_bis updated the task description. (Show Details)Apr 2 2015, 10:53 AM
aripstra added a comment.EditedApr 2 2015, 5:25 PM

In order to gather in context feedback on functionality in the past, we have had feedback links that take users directly to a survey tool (for example the feedback survey link in media viewer - with survey monkey). If we have to ask people to go to a wiki page and explain that we are taking them to another place for them to fill out the survey, the amount of fast, in context feedback we get will be a lot less than if we can just take people to the survey directly.

All that said, I am totally in favor of using open source tools when they are available and functional for the way we need to use them.
I have looked at Lime and as far as I can tell it is not nearly as advanced a survey tool as Qualtrics, and it is labor intensive to set up and use. I do not have engineering resources to build Lime to the place we need it to be to be an effective tool for the user research team. If that changes, and someone can provide those resources, I would be very happy to use Lime as it is open source. Until then, we need to use a survey tool that provides the functionality and usability we need right now. We want to understand the experience of, and gather feedback on functionality from all of the users. Many of our users (many readers and new / casual editors) are not familiar with the internal wiki feedback mechanisms, so we are attempting to reach out more broadly to gather a wider range of feedback. These types of tools help us do that.

ori added a subscriber: Moushira.May 23 2015, 2:07 PM

We should figure out the "labor intensive to use" part, but the setup part can be fixed. :) We're trying to start working on it tomorrow, thanks to Ori's help.

Change 213579 had a related patch set uploaded (by Nemo bis):
[WIP] Stub LimeSurvey configuration

https://gerrit.wikimedia.org/r/213579

Qgil added a subscriber: Qgil.

FYI - Legal has vetted Qualtrics and it has gone through the Contracts team.

Since this task is about "Identify survey services compatible with our privacy policy", we can add Qualtrics to the list.

If I remember correctly, Community-Relations-Support has used SurveyMonkey, and Developer-Advocacy uses Google Forms for i.e. event surveys. Are these services compatible with our privacy policy?

Principles that could guide next steps:

  1. Assess the compatibility of the survey services used by the WMF with our policies.
  2. Stop using survey services that are not compatible with our policies.
  3. Consolidate the use of different services in one (exceptions should be possible, with a justification).
  4. Investigate open source alternatives and propose and go for one if the feature set, sustainability of the project, and maintenance by Operations is clear.

Fabrice used SurveyMonkey for the Media Viewer survey, and I believe that it's been used by other projects in the past.

My team tested AllOurIdeas at the end of 2014. You can read a summary of its advantages and disadvantages here: https://meta.wikimedia.org/wiki/Community_Engagement_(Product)/Product_Surveys#Quarter_2_testing_phase

Qgil added a comment.EditedJun 4 2015, 10:43 AM

We are trying to help all open tasks listed under "Work continues after Lyon" at the Wikimedia Hackathon 2015 workboard finding their best way forward.

  • If you are participating in Wikimania, consider adding the #Wikimania-Hackathon-2015 project to get this task in that loop, which is about to start.
  • If you think this project could welcome help from a dedicated Google Summer of Code or Outreachy intern, or from an Individual Engagement Grant, add the Possible-Tech-Projects project.
  • If you would like to receive some other type of support (organizing a Tech Talk, establishing contacts with existing developer teams in Wikimedia or elsewhere, travel sponsorship for a related activity... you name it), please create a subtask explaining your request and associate it with Developer-Advocacy (or you can start by commenting here if you prefer).
  • Keeping the description, priority and assigned fields up to date always helps. :)

For some context about this message, see T101151: Evaluate which projects showcased at the Wikimedia Hackathon should be supported further. It is the last communication related to Wikimedia-Hackathon-2015 that we will post here.

fwiw, survey by the Design team at https://wikimedia.qualtrics.com/jfe/form/SV_e9zX99TJlwGVYMt

It says at the top:

Your privacy is important to us. We will only share your responses with Wikimedia staff and contractors and in accordance with our privacy policy. We will only publicly share your responses or survey results in aggregate or anonymized form. By answering these questions, you permit us to record and indefinitely retain your responses for research purposes.

Qgil updated the task description. (Show Details)Jun 9 2015, 7:45 AM
Elitre added a subscriber: Elitre.Jun 22 2015, 8:49 AM
Nemo_bis updated the task description. (Show Details)Jul 4 2015, 6:38 AM

I see I never responded to this - pardon!

AllOurIdeas has been used by Liaisons, and may be used again. I had it cleared by Legal, but if there are concerns about it, do let me know.

Qgil updated the task description. (Show Details)Jul 28 2015, 9:42 AM

So we have Qualtrics and AllOurIdeas that have been explicitly vetter by WMF-Legal. Is this enough to consider this task resolved, or what is missing?

For some (smaller) purposes, T89970: Enable microsurveys for long-term tracking of editing experience would solve the privacy problem: if we do it all in-house, then it's all subject to our own privacy policy and practices.

Also, informal on-wiki surveys are also possible. However, when my team offered the same survey both on-wiki and via Qualtrics last year, fewer than 1% of the respondents chose to reply on wiki.

The second point is very valuable @Whatamidoing-WMF.

@Qgil Manprit from Legal is looking at Google forms for surveys. Let's keep this task open at least until Legal is done with vetting (or not) Google forms. I'll update here what the result is.

This comment was removed by leila.

Thank you! @Rfarrand, Developer-Advocacy has been using Google Forms regularly. Let's try to find better alternatives like the ones described above the next time we need a registration/survey.

Qgil updated the task description. (Show Details)Aug 7 2015, 7:55 AM
leila added a comment.Aug 7 2015, 4:56 PM
This comment was removed by leila.

ECT will probably (mostly) move to Qualtrics soon as well, it is likely that we will stick with google forms for WMF internal event sign up and polling. @JAnstee_WMF from Learning and Evaluation recommends it highly and will be training me a bit on its use.
Any discussions about WMF surveys should probably include her as well! Adding her to the task. :)

Looking at the description of this task, I would say that we have almost completed it. We need to document its conclusions in a better place, though. Where?

https://meta.wikimedia.org/wiki/Survey_best_practices ?

Qgil moved this task from Backlog to Doing on the DevRel-September-2015 board.Sep 1 2015, 12:46 PM
egalvezwmf moved this task from Backlog to Survey Tools on the Surveys board.Sep 5 2015, 4:28 PM

I am interested in using an alternative service for more complex user surveys - Qualtrics is great but is expensive, so we are looking for other avenues.

Is Google Forms ok for user surveys? Is Lime Survey still being vetted?

Any updates on those points is appreciated!

@pizzzacat: if you need legal to review for a survey conducted by WMF, please email me directly and we can vet Google Forms or Live Survey. Thanks!

Qgil closed this task as Resolved.Sep 16 2015, 1:13 PM
Qgil claimed this task.

Looking at the description of this task, I would say that we have almost completed it. We need to document its conclusions in a better place, though. Where?
https://meta.wikimedia.org/wiki/Survey_best_practices ?

Copied at https://meta.wikimedia.org/w/index.php?title=Survey_best_practices&type=revision&diff=13657831&oldid=12023153

It looks like this task can be resolved., Thank you everybody!

I'm working on gathering resources for surveys, including providing comparisons across different survey software available that we have experience with. Will post link here when its ready.

@Slaporte I'm terrible at Phabricator and just saw this. Thank you, I will need review on a survey in a week or two.

Given the end of Safe Harbor in few weeks from now (http://www.europarl.europa.eu/sides/getDoc.do?pubRef=-//EP//TEXT+TA+P8-TA-2015-0388+0+DOC+XML+V0//EN ), the legal considerations are soon outdated. For instance, the current harassment survey banner links a USA corporation without warning EU users that they will be transferring private data to the USA.

Upon clicking the survey, the participant should be greeted with a legal disclaimer that includes the language "By answering our questions, you consent to the transfer of your responses to the United States and other places as may be necessary to carry out the objectives of this project." and gives the option of exiting the survey if the participant is not ok with such a transfer. Hope this helps.

Restricted Application added a subscriber: JEumerus. · View Herald TranscriptAug 2 2016, 1:57 PM
Nemo_bis updated the task description. (Show Details)Aug 2 2016, 1:59 PM
jayvdb updated the task description. (Show Details)Aug 2 2016, 2:15 PM

Just became aware of this task and thought I’d add a little information that I hope will be useful. The task is right - both Qualtrics and allourideas.org have been vetted by Legal. When we review a service in a circumstance like this, we go over their privacy policy, terms of service, and any individual contract we have with the third party in question, to make sure that the protections they offer user data are largely in line with our own. However, we also recognize that we have no control over how they may conduct themselves at all times and under all circumstances. Therefore, in the survey context, when we conduct a survey using a third-party service, we provide notice to survey-takers that the survey is on a third-party website, and we also provide links to the third party’s terms of service and privacy policy. When volunteers run a survey using a WMF-associated account, such as through Qualtrics, we ask them to provide the same notice and links. That way, survey takers can be informed of where/how their information is being handled.

Right now, Legal is working on updates to the survey toolbox on Meta, which will include some basic information about conducting surveys, including the use of third-party services. I will post to this ticket when the toolbox is complete, so all interested parties can take a look.

Nemo_bis updated the task description. (Show Details)Aug 3 2016, 7:44 AM
jayvdb updated the task description. (Show Details)Aug 3 2016, 8:28 AM
Nemo_bis updated the task description. (Show Details)Mar 8 2017, 2:45 PM
Lyzzy added a subscriber: Lyzzy.Jun 25 2017, 4:25 PM
Reedy changed the status of subtask T109606: Re-evaluate Limesurvey from Open to Stalled.Sep 11 2018, 7:39 PM