Page MenuHomePhabricator

Be able to choose an appropriate response format for a microsurvey question
Open, MediumPublic

Description

When product managers are setting up a microsurvey question, they need to be able to choose an appropriate response format. These three options (with possible examples of how to implement that type) are probably desirable:

  1. Binary choice:
    • Yes/no radio buttons
    • Thumbs up/thumbs down
  2. Likert scale ranges
    • "On a scale of 1 to 10..." or
    • Strongly agree/agree/neutral/disagree/strongly disagree
    • Smiley face scale
  3. Pick answer from list
    • e.g., Please set my user interface language to (long drop-down menu)
    • e.g., Please set my prefs to say He edits/She edits/They edit

Response options should not include free-form response fields. If free-form responses are wanted, then the participant should be given a link to an appropriate wiki page. For example:

  • Question, shown to editors with 1 to 50 edits at xxwiki: "Do you feel like you need more help with editing?"
  • Response options: Yes/No (plus the ever-present cancel/skip survey button)
  • Result: If yes, provide a link to the Teahouse or Help Desk pages (or to help pages). If no, wish them happy editing and get out of their way.
  • Product manager sees: 35% of newer editors at xxwiki said that they felt like they needed more help with editing.

Event Timeline

Whatamidoing-WMF triaged this task as Medium priority.Jan 2 2018, 6:57 PM
Whatamidoing-WMF created this task.

Why are free form fields excluded? (a link or so to a discussion on this or so would be nice)

I often have the impression that I lack qualitative data, and free form fields could be a way to collect this easily, whereas editing a wikipage might seem be a normal/easy thing, but still is a new context and activity (even more so for not-yet-very-experienced people)

Why are free form fields excluded? (a link or so to a discussion on this or so would be nice)

Because of the costs. As soon as you say "Here's an empty field, and you can type whatever you want in it", then you have to either keep the data hidden (which is not transparent) or you have to spend time removing personally identifying information (which is expensive – volunteer time, especially for people who understand privacy, is extremely valuable).

The burden created by free-form fields is one of the reasons that http://mediawiki.org/wiki/AFTv5 was killed.

Also, the idea is to collect data and have the results visible to everyone, immediately. Something like "22% said yes, 33% said no, 45% skipped the question" could be visible immediately. Free-form comments cannot be summarized automatically.

If you are concerned about people not being able to figure out how to edit an existing wiki talk page (many new editors struggle with that), then I can think of three alternatives:

  • using the Feedback extension,
  • using a Flow/Structured discussion page (available on mediawiki.org), or
  • using a &section=new URL, possibly with a preload template (used by thousands of contributors during the last few major discussions on Meta).

@Whatamidoing-WMF: Thanks, the explanation makes sense. Remark regarding the alternatives: I looked at the Feedback-Ext, it seems to be stopped.

Because of the costs. As soon as you say "Here's an empty field, and you can type whatever you want in it", then you have to either keep the data hidden (which is not transparent) or you have to spend time removing personally identifying information (which is expensive – volunteer time, especially for people who understand privacy, is extremely valuable).
The burden created by free-form fields is one of the reasons that http://mediawiki.org/wiki/AFTv5 was killed.

Keep in mind that the ability to collect free-form responses is essential for any research project which uses a grounded theory approach, like the "Why we read Wikipedia" research. We do already have tools for this (that study used Google Forms, possibly coupled with the Quick Surveys component), but don't forget that the need is very real.

Also, the idea is to collect data and have the results visible to everyone, immediately. Something like "22% said yes, 33% said no, 45% skipped the question" could be visible immediately. Free-form comments cannot be summarized automatically.
If you are concerned about people not being able to figure out how to edit an existing wiki talk page (many new editors struggle with that), then I can think of three alternatives:

  • using the Feedback extension,
  • using a Flow/Structured discussion page (available on mediawiki.org), or
  • using a &section=new URL, possibly with a preload template (used by thousands of contributors during the last few major discussions on Meta).

Flow pages and section=new links are nice, but they don't help with the the core dilemma of dealing with free-form responses that you identified. I think labor-intensive, after-the-fact cleanup is totally unworkable, both because of the time burden and because of the risk to privacy before the cleanup can happen. So we're left with the unfortunate reality that lack of transparency is an essential part of many, many different research projects. For example, the full data sets from the CE Insights surveys will almost certainly never be released publicly, because of the risk to privacy.

So I feel like the choice here is really between not supporting free-form answers at all, or support storing them privately for researcher-only access. Either one can be defended, but I'm skeptical that there's a middle ground. Admittedly, we do take a middle-ground position for user-submitted content on the projects in general (allowing free posting and cleaning up later if necessary), but I would be very hesitant to extend that to research projects.

Jan_Dittrich added a comment.EditedJan 10 2018, 9:46 AM

So I feel like the choice here is really between not supporting free-form answers at all, or support storing them privately for researcher-only access.

Researcher-only access is very common, but it might not fit the way this is intended to work.

One alternative is making clear that the answers will be public (but do not necessarily publicly be linked to an account). People post publicly on wiki or phabricator all the time (linked to their account and not checked by anybody) so as long as this it is very transparent that it will be public it might be a good middle ground (Yes, it also creates bias, but this is a concern that needs to be reflected in the research imho)

@Capt_Swing: What do you think?

However: What about moving this to a sub-issue to focus each conversation?

Jan is correct: This is not how I intend for this product to work. But it's worth talking about, so I've created T184645: Consider ways to handle free-form text responses for the discussion. Please "Be bold" and expand the task description.

matmarex added a subscriber: matmarex.

Removing MediaWiki-Page-editing so that this doesn't clutter searches related to actually editing pages in MediaWiki. This project should only be on the parent task, probably (T89970). [batch edit]

matmarex removed a subscriber: matmarex.Jan 25 2018, 2:17 AM