Page MenuHomePhabricator

Send out browser testing user satisfaction survey
Closed, ResolvedPublic

Description

Form

https://docs.google.com/forms/d/1g1E_S-kLCO8E7wsqtVkHzzmAYW36Ml6hkOZhVH9QtKU/formResponse

Goal

To get a baseline of user satisfaction in our browser-testing infrastructure/tooling.

Target audience

Notes

Testing: Where does it hurt? video might be good inspiration for questions.

Scale

5 level scale might be enough for our purpose

  • :'( 1
  • :( 2
  • :| 3
  • :) 4
  • :D 5

Landing page

  • Thank you for taking the time for this survey. It should take you up to 5 minutes. Most of the questions have simple 5 level linear scale. There are 5 sections, and the last question in each section will be free form text field, so you can leave comments on anything we forgot to ask.

Jenkins

  • Gerrit triggered Selenium+Jenkins jobs (mwext-mw-selenium, after patch set submission).
    • Jobs are stable enough.
    • Jobs are fast enough.
  • Time triggered Selenium+Jenkins jobs (selenium*, daily)
    • Jobs are stable enough.
    • Jobs are fast enough.
  • I know how to use CI entry points

Ruby

JavaScript

  • How do you feel about...

Selenium

  • How do you feel about...
    • writing tests
    • fixing failed tests

Getting help

  • Rate your experience getting help via the below methods.
  • How do you feel about...
  • I need help with getting started with testing
    • I would like to pair with somebody
    • I would like to attend on-line workshop
    • I would like to attend in person workshop (hackathon, conference)
  • I need help with fixing failed tests
    • I would like to pair with somebody
    • I would like to attend on-line workshop
    • I would like to attend in person workshop (hackathon, conference)

Event Timeline

greg created this task.Mar 28 2016, 11:57 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 28 2016, 11:57 PM
greg added a comment.Apr 4 2016, 8:24 PM

I emailed Edward asking about survey infra options.

greg added a comment.Apr 4 2016, 9:19 PM

Response: Qualtrics is the only thing that we have an account with (at least that Edward's team has an account with). He suggested google forms can be good if done well.

zeljkofilipin updated the task description. (Show Details)Jun 2 2016, 3:25 PM
zeljkofilipin updated the task description. (Show Details)
zeljkofilipin updated the task description. (Show Details)Jun 2 2016, 3:31 PM
zeljkofilipin renamed this task from Send out browser testing user satisfaction survey to Browser testing user satisfaction survey.Jun 3 2016, 1:43 PM
zeljkofilipin updated the task description. (Show Details)
zeljkofilipin moved this task from Next 🔜 to In Progress 🔨 on the User-zeljkofilipin board.
zeljkofilipin moved this task from Backlog to Active on the Surveys board.
zeljkofilipin updated the task description. (Show Details)Jun 3 2016, 1:50 PM
zeljkofilipin updated the task description. (Show Details)Jun 3 2016, 1:55 PM
zeljkofilipin updated the task description. (Show Details)Jun 3 2016, 1:59 PM

I have asked QA mailing list for comments.

greg updated the task description. (Show Details)Jun 6 2016, 4:46 PM
zeljkofilipin updated the task description. (Show Details)Jun 6 2016, 4:47 PM
greg updated the task description. (Show Details)Jun 6 2016, 8:27 PM

Did another pass, left a XXX regarding the "getting started" documentation question (which one? link to it).

  • Free text

I would be great to include some free text box fields to provide context about some
answers. For instance, personally "I would like to help with fixing failed tests" AND
"I would like to pair with somebody" BUT.. there's a but (time zones and scheduling are proving troublesome for me) :-)

  • Consider using the word "would" in your questions.

"I prefer writing end-to-end ('browser') tests using this language"
This is a strange question for me. I would prefer writing browser tests in JS but I have never written browser tests in JS.

  • Also consider asking questions around platform stability / infrastructure.
  • Free text

I would be great to include some free text box fields to provide context about some
answers. For instance, personally "I would like to help with fixing failed tests" AND
"I would like to pair with somebody" BUT.. there's a but (time zones and scheduling are proving troublesome for me) :-)

Are you saying every question should have a free form answer? Or just some of the questions? Which ones?

  • Consider using the word "would" in your questions.

I am not sure what you mean.

"I prefer writing end-to-end ('browser') tests using this language"
This is a strange question for me. I would prefer writing browser tests in JS but I have never written browser tests in JS.

We (Release-Engineering-Team) would provide tools and training, like we did for Ruby.

  • Also consider asking questions around platform stability / infrastructure.

There are a few questions like that:

  • Gerrit triggered Selenium+Jenkins jobs (after patch set submission)
    • are stable enough
    • are fast enough
  • Time triggered Selenium+Jenkins jobs (daily)
    • are stable enough
    • are fast enough

Can you provide example(s)?

zeljkofilipin updated the task description. (Show Details)
zeljkofilipin updated the task description. (Show Details)
zeljkofilipin updated the task description. (Show Details)
zeljkofilipin updated the task description. (Show Details)
zeljkofilipin updated the task description. (Show Details)
zeljkofilipin updated the task description. (Show Details)

The form is created. Please take a look and let me know if anything needs to be fixed. If you want to edit the form yourself, let me know and I will give you access.

greg added a comment.Jun 14 2016, 8:17 PM

nitpick/bikeshed: I'm actually thinking that a 5-point scale (vs 3) would be better. 3 is nice and quick for people but 5 gives them a little more wiggle room and, I believe, will increase the chances that we see change (good or bad) in a follow up survey. We used a 7-point scale for the MW-Vagrant survey before: https://www.mediawiki.org/wiki/MediaWiki-Vagrant/Surveys/2014Q4

@greg agreed, changed to 5-point scale.

I was browsing the surveys tag and saw this one. Some suggestions/thoughts:

  • Use the grid option in google forms; you can combine more than one question this way
  • Check out these response scales. The smileys could be misleading/confusing. especially for the "I like.." questions; this could create bias in your responses. You could use the google form number scales, but its better to anchor each number with a word. If you want to still use smileys, you should change the questions to: "How do you feel about...." instead of "I like". You should probably also use upright smileys rather than sideways ones...
This comment was removed by zeljkofilipin.

I was browsing the surveys tag and saw this one. Some suggestions/thoughts:

Thanks @egalvezwmf!

  • Use the grid option in google forms; you can combine more than one question this way

Thanks, I was not aware of this feature.

  • Check out these response scales. The smileys could be misleading/confusing. especially for the "I like.." questions; this could create bias in your responses. You could use the google form number scales, but its better to anchor each number with a word. If you want to still use smileys, you should change the questions to: "How do you feel about...." instead of "I like".

Thanks, I will think about it.

You should probably also use upright smileys rather than sideways ones...

Should I copy/paste unicode characters from here? http://unicode.org/emoji/charts/full-emoji-list.html

Is there a way to test support for characters? Greg's machine did not display a couple correctly.

zeljkofilipin updated the task description. (Show Details)
zeljkofilipin updated the task description. (Show Details)
zeljkofilipin updated the task description. (Show Details)

Ok, I think I have done everything suggested. I have used sideways emoji (:)) instead of upright () because I did not have the time to test how will unicode emoji display on various platforms. At least @greg had trouble with displaying some unicode emoji.

As far as I am concerned, we can launch the survey. We might wait a week since next week is wikimania.

greg added a comment.Jun 17 2016, 8:07 PM

As far as I am concerned, we can launch the survey. We might wait a week since next week is wikimania.

At this point, yeah, that makes sense.

  • Free text

I would be great to include some free text box fields to provide context about some
answers. For instance, personally "I would like to help with fixing failed tests" AND
"I would like to pair with somebody" BUT.. there's a but (time zones and scheduling are proving troublesome for me) :-)

Are you saying every question should have a free form answer? Or just some of the questions? Which ones?

The more the better (all optional). I would argue the more open ended your questions are the better the feedback you'd get.
But yes definitely for this question (any further comments/explain your answer).

  • Consider using the word "would" in your questions.

I am not sure what you mean.

Compare:

  1. "I prefer writing end-to-end ('browser') tests using this language"
  2. "I would prefer writing end-to-end ('browser') tests using this language"

My answers for these would be different. I have never written browser tests in JS. I would probably prefer it but the first point suggests that I have experience. My suggestion is to be more careful with the wording here.

The more the better (all optional). I would argue the more open ended your questions are the better the feedback you'd get.
But yes definitely for this question (any further comments/explain your answer).

Every section now has optional comments.

Compare:

  1. "I prefer writing end-to-end ('browser') tests using this language"
  2. "I would prefer writing end-to-end ('browser') tests using this language"

My answers for these would be different. I have never written browser tests in JS. I would probably prefer it but the first point suggests that I have experience. My suggestion is to be more careful with the wording here.

Good point. I will change the questions.

The survey is sent.

Hi @zeljkofilipin. Could you add this survey to the community engagement calendar? https://meta.wikimedia.org/wiki/Community_Engagement/Calendar

Thanks, J

Risker added a subscriber: Risker.Jul 2 2016, 3:55 AM

What's the privacy policy for this survey?

greg added a comment.Jul 2 2016, 6:19 PM

Good question, Risker.

All submissions will be seen and collated by members of the Release Engineering team (namely Zeljko and I).

Submissions do not require logging into Google nor providing a name or email.

A summarized version will be made public (on mw.org). The 1-5 ratings will be averaged and shared. The free form comments will not be shared directly (iow: no direct quotes but instead paraphrasing one or more at a time).

Does that help?

I or Zeljko will send out a reply with this above clarification on Monday (a US holiday, so unlikely) or Tuesday (more likely). I'm going to go back to my birthday today :)

greg renamed this task from Browser testing user satisfaction survey to Send out browser testing user satisfaction survey.Jul 2 2016, 6:21 PM
greg closed this task as Resolved.Jul 2 2016, 6:23 PM

This task is resolved. Follow up/summarization tracked in T139247.

What's the privacy policy for this survey?

Apologies for forgetting about that, this is my first survey. The privacy policy is here: https://wikimediafoundation.org/wiki/Survey_Privacy_Statement_for_July_2016_Browser_Testing_Satisfaction_Survey