Page MenuHomePhabricator

First QuickSurvey for reader segmentation research - external survey
Closed, ResolvedPublic0 Story Points

Description

As a member of analytics I would like to enable the following survey on English Wikipedia:
https://gerrit.wikimedia.org/r/251133

Background:
We want to:

run a survey on English Wikipedia on Desktop for x% of the traffic until we receive n responses. (we will specify x and n). The survey will have one question and that is: "Why are you reading this article today?". The question is chosen carefully to address the needs of this first survey. The participant will be provided a free-form text box to enter the reason.

We will do manual categorization of responses after the first survey to identify top level categories that the readers signal as reasons for reading a specific article. This survey is designed with the input from Capt_Swing, Cervisiarius, ellery, and leila.

This is the first survey in a series of surveys. The next one will include the categories identified in the first survey and will use those as labels offered to the participant. That information then will be used to analyze the logs in light of that new information.

array(
		"@description"=> "description of the survey",
		"description" => "Reader-segmentation-1-description",

		"@link"=> "external link to the survey (must be https)",
		"link"=> "https://docs.google.com/forms/d/17vKKxhVF4A3iO71JNQ4xbvc42Md3U87WgnYwmXISqoM/viewform",


		"@question"=> "survey question message key",
		"question"=> "Reader-segmentation-1-message",

		"@privacyPolicy"=> "message key for privacy policy. May contain links.",
		"privacyPolicy"=> "Reader-segmentation-1-privacy",

		"@coverage"=> "percentage of users that will see the survey",
		"coverage"=> "0.005",

		"@platform"=> "for each platform (desktop, mobile), which version of it is targeted (stable, beta, alpha)",
		"platform"=> array(
			"desktop"=> ["stable"],
			"mobile"=> []
		)
	)

Details

Related Gerrit Patches:

Related Objects

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

@leila this is so great! Thank you for taking this on. Do you have everything you need to make this happen?

@aripstra added this information, too:
https://meta.wikimedia.org/wiki/Research:Global_South_User_Survey_2014
if you go to page 194, in the deck on this page, you can see the questions that options that were provided in that survey for a similar question:
"How do you use wikimedia projects?"(for "all users" the percentages of respondents reporting they use the wiki for that purpose are noted next to the option

Thanks for sharing this. We do not want to use the term "use" and we want to stay focused on understanding possible reasons for "reading", "article", and "today". We decided that "use" directs to a user to a specific set of responses, and as much as possible we want to avoid directing them to any specific direction.

leila added a comment.Oct 14 2015, 5:26 AM

@Jdlrobson can we collect free form (text entry) answers from readers in QuickSurveys?

Nope this wasn't build into the original requirements it would need to be added. :/

leila added a comment.Oct 14 2015, 3:37 PM

@Jdlrobson, got it. I made a ticket for it: T115485

@atgo the global south user survey questions are asking a different, but related question, which is "what is your external trigger for using WP? (School, work, personal...) The question I think we need to ask first is "what are you here to accomplish?" Regardless of whether it is for school or personal, from a product perspective, we are much more concerned with whether or not it is a quick fact or to learn about a subject.

@leila @Jdlrobson @atgo In the interest of time, I think we should hold on freeform for now. For now let's use "other" as a multiple choice only (no additional text). If we find that a sig % of users are clicking it, we can decide to build free text.

^ from a product-building/design/eng perspective, we are dying to know what the breakdown of primary use cases are ("is it 90% quick look up or 10%"). Far less interested in a comprehensive list of things people use it for and fairly confident we have the primary ones. If we don't we will see a lot of 'others' being clicked.

We're going to run a multiple choice as the first survey and let this deeper dive come a bit later once the feature has been used a bit. With that in mind, I'm making some changes to the description here. @leila if you want to capture the rest of this somewhere else, let's make another task. Also if you want to remove Research for your workflow, feel free.

leila added a comment.Oct 15 2015, 1:33 AM

@atgo, I think the tasks are being confused. This task is assigned to me and it's a Research-and-Data task. If the Reading team wants to run another survey, please make a separate task for that. I'll make the title of this task clearer.

leila renamed this task from Design first QuickSurvey to Design first QuickSurvey for reader segmentation research.Oct 15 2015, 1:33 AM
leila added a comment.Oct 15 2015, 1:37 AM

@JKatzWMF I'll keep this task focused on the research question we, Research, want to address, i.e., building a model that can tell us what the purpose of visit for every pageview/unique client is. It looks like the Reading team wants to run a survey to get answer to a similar question, but not the exact same one. Please make a separate task for that survey.

Just as a comment: I can understand both Leila's preference for doing a full systematic gathering and coding of free-form responses first, and Jon's preference for moving quickly (separately, I would argue that it's beneficial to keep the first ever use of the new QuickSurvey feature simple, as there might be enough complications from bugs and other unforeseen aspects).

But what I don't understand is why T115485 should be regarded as a blocking task in case it is decided that R&D will conduct a survey that is based on gathering free-form responses. There are other options for doing that which have been used frequently over the years and are well-tested. Qualtrics in combination with a CentralNotice banner has been used for the 2014 Global South Survey that Anne mentioned above and also e.g. for the 2012 Editor Survey which involved in-depth analyses of two free-form questions (see here and here - the latter is actually pretty much an editor analogon of the question posed to readers here). I'm sure @egalvezwmf could advise on this and other options if asked.

leila added a comment.Oct 15 2015, 5:52 PM

@Tbayer, good point. The reason I'd like to keep T115485 as a blocker for this task is that although in theory R&D can run the first survey with Qualtrics, the second survey needs to happen in QuickSurveys, because we need to match the responses of that survey to logs. In reality T115485 is blocking the second survey (for which we don't have a ticket yet) and if we cannot do that survey, that ticket becomes the blocker for the first survey because we don't want to do the first one unless we know we can do the second one. :-)

I've talked to Jon and Toby about it and I'm waiting for their response on whether we can have free form text in 2-3 weeks. If we can, Qualtrics for the first survey works fine, and I'll remove the blocker for this task.

Thanks Leila. Removed this as a blocking task and made another for my
purposes, T115647

This makes much more sense to me and its nice to have this clarity between the work streams of "Product" and "Research". I'm noticing we are missing "Documentation" & "Learning" for both the process used in developing the survey and for using the new tool, which I'd be happy to facilitate.

leila added a comment.Oct 16 2015, 4:41 PM

@egalvezwmf good point. Research survey will be documented in: https://meta.wikimedia.org/wiki/Research:Index We will follow the usual practice there. Explain the survey, run it, update the statistics. I'll update this ticket once/if we have a confirmation that we're doing this task.

leila moved this task from Staged to Blocked on the Research board.Oct 22 2015, 5:10 AM
atgo renamed this task from Design first QuickSurvey for reader segmentation research to First QuickSurvey for reader segmentation research - external survey with free text entry field.Oct 23 2015, 8:47 PM
leila added a comment.Oct 27 2015, 4:48 PM

@atgo is there an easy way for the Qualtrics survey to open as a window in the same browser the user is in or the user will have to be taken to a Qualtrics survey page in the same or a new tab? I need this information to finalize the question in this task.

leila added a comment.Oct 27 2015, 4:50 PM

@atgo I'm not sure why T110661 is a blocking task for the current task.

@atgo is there an easy way for the Qualtrics survey to open as a window in the same browser the user is in or the user will have to be taken to a Qualtrics survey page in the same or a new tab? I need this information to finalize the question in this task.

My assumption (@Jdlrobson please correct me if I'm wrong) is that the survey would open in a separate tab so that the reader could easily close the tab and navigate back to the page they were previously viewing.

@atgo I'm not sure why T110661 is a blocking task for the current task.

we need to deploy the feature for you to use it... again, @Jdlrobson correct me if I'm wrong here

atgo updated the task description. (Show Details)Nov 3 2015, 12:57 AM
atgo updated the task description. (Show Details)Nov 3 2015, 1:07 AM

I've added the array with the pieces you need to the description. @leila will update them with the text.

It may take a bit more time to get the link to the privacy policy, will keep you posted.

leila added a comment.Nov 3 2015, 5:14 PM

@Jdlrobson: how do we do traffic sampling? Is it session based, or request based. For example, if Coverage=1, does it mean that 1 out of every 100 requests will see the survey or does it mean 1 out of every 100 session will see the survey?

In a perfect world the latter but it relies on local storage to store a session so it might be skewed somewhat by people with full local storage or older browsers.

leila added a comment.Nov 3 2015, 8:07 PM

Thank you, @Jdlrobson. Working on the array.

leila added a comment.Nov 4 2015, 1:51 AM

@Jdlrobson we may need a little bit more time for finalizing the parameters in array. What is the deadline for letting you know the details?

Jdlrobson renamed this task from First QuickSurvey for reader segmentation research - external survey with free text entry field to First QuickSurvey for reader segmentation research - external survey.Nov 4 2015, 5:46 AM

Thursday latest (see mail "Deploying QuickSurveys")

Change 251133 had a related patch set uploaded (by Jdlrobson):
WIP: First QuickSurvey for reader segmentation research - external survey

https://gerrit.wikimedia.org/r/251133

leila added a comment.Nov 4 2015, 10:54 PM

@Jdlrobson, per email discussions, we want the survey to run in enwiki. I'll update the description as well.

leila updated the task description. (Show Details)Nov 4 2015, 10:55 PM
leila updated the task description. (Show Details)Nov 6 2015, 5:46 PM
leila updated the task description. (Show Details)
leila added a comment.Nov 6 2015, 5:55 PM

@Jdlrobson I have entered the details in the array. (the 0.002 is in percentage, i.e., we want to sample 2 out of 100K).

@atgo, I see a discussion in https://meta.wikimedia.org/wiki/Meta:Requests_for_help_from_a_sysop_or_bureaucrat#Re:_Survey_powered_by_Qualtrics about the wording around the use of external Survey tools. Based on Nemo's recommendation there, I suggest that we go with: "Survey data handled by a third party." and then add the privacy policy link in the array. I'm pinging you since this comment will apply to future surveys on QuickSurvey if external tools are used as well.

There are a few issues with this.

  • Firstly the question etc should be message keys - you'll need to edit message interface message on enwiki with your staff account to set these up.
  • The privacy message is not just a URL it should also have text.
  • Are you sure you don't want a description? If not you'll need to setup a blank description message anyway.

Looks like this right now:

leila updated the task description. (Show Details)Nov 6 2015, 7:29 PM
leila updated the task description. (Show Details)Nov 6 2015, 8:38 PM
leila added a comment.Nov 6 2015, 8:40 PM

@Jdlrobson Turned out I couldn't get Staff rights so quickly, and Guillaume helped out. Please check the description and let me know if anything is missing. The keys are from https://en.wikipedia.org/wiki/MediaWiki.

What are the keys?

leila added a comment.Nov 6 2015, 8:59 PM

I put them in the Description part of this ticket, @Jdlrobson.

Done.


Over to @jhobs for SWAT deploy.

Jdlrobson removed leila as the assignee of this task.Nov 6 2015, 9:34 PM
Jdlrobson updated the task description. (Show Details)
Jdlrobson edited a custom field.
Jdlrobson added a subscriber: leila.

@leila (or @atgo), can you confirm that this is the correct link for the survey and please update this card and/or ping me when it's been enabled? https://docs.google.com/a/wikimedia.org/forms/d/17vKKxhVF4A3iO71JNQ4xbvc42Md3U87W%20gnYwmXISqoM/viewform

Thanks!

leila added a comment.Nov 10 2015, 6:27 PM

@leila (or @atgo), can you confirm that this is the correct link for the survey and please update this card and/or ping me when it's been enabled? https://docs.google.com/a/wikimedia.org/forms/d/17vKKxhVF4A3iO71JNQ4xbvc42Md3U87W%20gnYwmXISqoM/viewform
Thanks!

@jhobs, the correct file is in the array in the description. When I click on your link, I don't get what I get from the link in Description which is: https://docs.google.com/forms/d/17vKKxhVF4A3iO71JNQ4xbvc42Md3U87WgnYwmXISqoM/viewform

Change 251133 merged by jenkins-bot:
First QuickSurvey for reader segmentation research - external survey

https://gerrit.wikimedia.org/r/251133

Change 252456 had a related patch set uploaded (by Jhobs):
Copy wmgQuickSurveysConfig to wgQuickSurveysConfig

https://gerrit.wikimedia.org/r/252456

Change 252456 merged by jenkins-bot:
Refactor wmgQuickSurveysConfig to wgQuickSurveysConfig

https://gerrit.wikimedia.org/r/252456

jhobs added a comment.EditedNov 12 2015, 1:03 AM

@leila We need to add another key for the link instead of hard-coding it into the config. Can you get Guillaume to help you create https://en.wikipedia.org/wiki/MediaWiki:Reader-segmentation-1-link with just the survey link as the contents by 4pm PST Thursday?

@leila We need to add another key for the link instead of hard-coding it into the config. Can you get Guillaume to help you create https://en.wikipedia.org/wiki/MediaWiki:Reader-segmentation-1-link with just the survey link as the contents by 4pm PST Thursday?

{{done}} :)

First survey deployed to enwiki successfully!

phuedx closed this task as Resolved.Nov 13 2015, 9:35 AM
phuedx added a subscriber: phuedx.

First survey deployed to enwiki successfully!