Page MenuHomePhabricator

Summarize results from the template prototype surveys
Closed, ResolvedPublic5 Estimated Story Points

Description

Write a Spark module to collect survey responses. Filter down to responses to our survey. Push limited demographic information into Spark to validate sampling. Decide how to sum choices, but the most basic method would be to calculate the percentage of individuals who voted for each choice.

Monitor total number of responses and changes in impression and response rate, decide when to end the surveys.

Omit responses submitted internally:

  • 2020-06-26T10:02:41Z
  • 2020-06-26T09:51:57Z

Event Timeline

awight created this task.Jul 1 2020, 2:44 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 1 2020, 2:44 PM
awight updated the task description. (Show Details)
awight set the point value for this task to 5.
awight removed the point value for this task.
awight updated the task description. (Show Details)Jul 1 2020, 2:56 PM
awight claimed this task.Jul 28 2020, 11:56 AM
awight moved this task from Backlog to In sprint on the WMDE-Templates-FocusArea board.
awight set the point value for this task to 5.

Data will be scrubbed after 90 days, so we should anonymize and copy to a less transient table if we might need to revisit these results later.

Looks good to me!

awight added a comment.Aug 5 2020, 7:45 AM

Note: the report has to be run manually, so we should decide on a refresh schedule.

I think it should be fine to just refresh it one more time before the meeting, so either on the 14th or 17th?

Lena_WMDE closed this task as Resolved.Aug 13 2020, 12:11 PM

@awight - forgot to mention, we also factored in/discussed these results in relation to priority and community interest. Thanks again for making the gitlab page.

awight added a comment.EditedAug 19 2020, 8:09 AM

@ECohen_WMDE to answer some of the questions which came up recently,

  • I can confirm that we have zero votes for "1f". There doesn't seem to be any problem with the answer ID, etc. . My only thought is that it came last in the list, and had an extremely short description with no picture. All of these factors would work against it. Also, at least in the English version, the description isn't very clear and no motivation is provided ("This is an effective way to find examples of templates used on similar articles.").
  • We had 14 total participants, 9 on dewiki and 5 on metawiki. These are not deduplicated across the two wikis, some people responded in both places.
  • These 14 participants answered 26 individual surveys, filling out 83 checkboxes.
  • The first response was on June 30 for metawiki and July 7 for dewiki, and we last refreshed the results on August 19th. During these date ranges, there were 314 pageviews to dewiki and 399 pageviews to metawiki. This gives a response rate of 2.9% (9 / 314) on dewiki and 1.3% (5 / 399) on metawiki.

Thanks so much! Very helpful. And you didn't refresh responses because there were no additional ones right?

Also I agree on the zero votes for 1f - I don't think it's necessarily a clear reflection on how people might feel about the feature but instead a reflection on it's presentation.

Also - is it possible to see the total number of participants, separate from the responses?

you didn't refresh responses because there were no additional ones right?

Only because I was being lazy, and partially because I felt that the long tail (at least 1 week with no additional responses) could be unfair to include in the response rate. But now I'm thinking, it's worse to "peek" at results and let that affect how analysis is done. I'll recalculate...
... and there's one more survey response in the mailbox as of Monday, thanks for making this suggestion!

The report is refreshed, and now includes a text table with total votes, in case you want to refresh your graphs as well. I could try to port them into the notebook but hesitated because the fidelity would suffer.

Also - is it possible to see the total number of participants, separate from the responses?

Good point, I was completely wrong about the response rate above! I'll update that comment in place, but the bottom line is that we had 9 participants through dewiki and 5 on metawiki. As it turns out there was a session ID included in each row, so this is an accurate estimate of unique individuals.

I still don't have a nice way to deduplicate users between the two groups, I found that the user's session ID is different across each site.

The report is refreshed, and now includes a text table with total votes, in case you want to refresh your graphs as well. I could try to port them into the notebook but hesitated because the fidelity would suffer.

Thanks! And glad to hear we have another response. I took a look at the report though and it doesn't look like any of the numbers have changed?

9 participants through dewiki and 5 on metawiki

This is also great news! Means we ~tripled the number of users we are hearing from through this. Glad there was a way to do this.