Page MenuHomePhabricator

Survey users about mediasearch on commons
Open, Needs TriagePublic

Description

Use QuickSurveys to gather evidence as to whether users regard mediasearch as an improvement over plain old Special:Search

Event Timeline

Cparle created this task.Jul 20 2020, 4:54 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 20 2020, 4:54 PM
Cparle added a subscriber: Isaac.Jul 20 2020, 4:54 PM
Cparle renamed this task from Survey users about mediasearch to Survey users about mediasearch on commons.Jul 20 2020, 5:01 PM

What is/are the question(s) we'd like to ask?

And who do we want to survey (what are the criteria the audience)? Any random commons visitor? Anons? Or experienced with over X contributions? Only visitors of Special:MediaSearch? Or visitors of one of SDC's talk pages?

@matthiasmullie I believe in previous discussions we talked about surveying only visitors of Special:MediaSearch (which we are about to start to promote more widely), and asking one question, something like "Is this search experience an improvement over the default search experience?", perhaps with optional room for comments.

@Ramsey-WMF and @mwilliams, would love your thoughts.

Hi! Based on our OKR planning, we have two questions to ask (but we don't need to ask them both at once).

  1. Is this search experience an improvement over the default Commons search experience?
  2. Do you think this search will lead you to start your image searches here on Commons, or will you use a major search engine (Google, Bing, etc.) as your image search starting point?

I wonder if @Isaac has thoughts on how we word those questions for optimal results :)

matthiasmullie added a comment.EditedSep 9 2020, 4:51 PM

Does this look ok?


A few things to be aware of, though, since QuickSurveys doesn't appear to have been developed for this kind of usage:

  • we can't target specific special pages (they all have the same id of 0), unless we work around that by explicitly loading it via an URL param
  • it doesn't remember whether you've already filled out the survey previously, so you'll continue to see the question (until we turn it off)
    • that's not really an issue for results, though - session data is recorded and can be filtered out

So, we would need to add ?quicksurvey=<survey name> to the link from wherever we want "advertise" this new media search. Is that possible?
If not, we'll need to go make more substantial changes in Extension:QuickSurveys.

Change 626338 had a related patch set uploaded (by Matthias Mullie; owner: Matthias Mullie):
[mediawiki/extensions/QuickSurveys@master] Allow QuickSurveys on special pages

https://gerrit.wikimedia.org/r/626338

nettrom_WMF moved this task from Triage to Tracking on the Product-Analytics board.
nettrom_WMF added a subscriber: nettrom_WMF.

@CBogen @Ramsey-WMF Does T258419#6447736 (screenshot and caveats) look acceptable for us? (If not, we're likely in for a sizeable refactor of QuickSurveys, or consider alternative solution)

The screenshot looks good to me - just hoping that @Isaac can help us verify the wording. Will we have to run two separate surveys to ask the two questions, or can we ask them both at once?

As far as the URL param, that's disappointing but seems like something we'll just have to work with.

I don't think continuing to see the survey is a big deal - not optimal but something we can live with.

@Ramsey-WMF do you agree?

mwilliams added a comment.EditedSep 18 2020, 3:50 PM

I'm a bit worried how the grid and quick view will work/feel with losing that much width to the survey. I would certainly respond negatively to a survey that made the functionality of the thing we are asking about worse. @matthiasmullie It sounds like there isn't much flexibility on the layout or design with this? Is it worth my while to consider a few other ways this could work?

Also, we don't have the second question Ramsey asked in that screenshot, which would take up even more space. Wondering if we use type: external on this.

Feel similarly about the survey continuing to be there even after you've responded, that feels pretty broken.

The results of this survey will heavily depend on how it is presented IMO.

Change 628785 had a related patch set uploaded (by Matthias Mullie; owner: Matthias Mullie):
[mediawiki/extensions/WikibaseMediaInfo@master] [WIP] Hacky way to load media search survey

https://gerrit.wikimedia.org/r/628785

QuickSurveys only allows 1 question, so we either run separate surveys, or so an external survey. (Since @Ramsey-WMF said we don't have to ask both at once, I assumed we'd do 2 separate?)

Had a quick chat with @mwilliams: while there is no flexibility in how the survey looks, we can attach it wherever we want. @mwilliams will check what (if anything) would be a better alternative.

So the 2 remaining downers are the ?quicksearch= url requirement for the survey to show up here, and the problem of it continuing to show up in that case (even if already completed).
It'll be a massive hack, but we can work around those. Given that it's only a short-lived survey, I can live with the hacky solution in this case.

mwilliams added a comment.EditedSep 21 2020, 5:01 PM

Thanks Matthias!

Working on alternatives on where to attach the survey...in the meantime a few questions to help me figure this out...maybe for @Ramsey-WMF?

  • Will 100% of Special:MediaSearch visits see the survey? If we want to ask two questions in two separate surveys, will the two survey's be split 50/50?
  • How long are we thinking of leaving this up and how many responses will feel "significant"?
  • Without it being the default for some users, most of the survey answers will be expert Common users who know where the link to Media Search is and actively test it. If that's true, couldn't we just put up the full survey on the project page to target the same people? Or put a link to the survey near where people access Media Search?
  • Based on the QuickSurvey's extension, we need set answers for the multiple choice selection. The questions are pretty binary, so should the possibilities map to what Matthias's has? Essentially "Yes, No, No preference"? I wonder if the second question, "Do you think this search will lead you to start your image searches here on Commons", needs answers that are a bit more nuanced. Something like "More often, less often, same" since I can't imagine us realistically being in the same ballpark as Google for all your image needs at this point and Yes / No doesn't cover that.
Isaac added a comment.Sep 24 2020, 5:16 PM

Thanks for the patience -- a few thoughts:

Is this search experience an improvement over the default Commons search experience?

Respondents tend to bias towards saying "yes" to questions (or conversely, saying "no" to questions) so it's best to avoid yes/no questions where you can: https://en.wikipedia.org/wiki/Response_bias#Acquiescence_bias
I'd reword to something more like, "Which search experience do you prefer?" with the answers something like "This version", "The old version", "No preference", and "Other" with an open-text field. The open-text option in particular I think is key if you want this to be more than a checkmark and something that potentially gives feedback you could incorporate into design as well. I'm also worried that many users won't remember what the old version is. The only fix I can really think of for this is that you instead run an A/B test where half the users get the old version and half get the new version and for each you ask them a Likert-scale question about satisfaction or something like that.

Do you think this search will lead you to start your image searches here on Commons, or will you use a major search engine (Google, Bing, etc.) as your image search starting point?

This one is really hard. It has challenges with acquiescence bias as well but I think more importantly, people are not good at predicting what their behavior will be. Personally, I think it took me at least ten tries to switch to DuckDuckGo and now I'm super happy with it but willingness to switch was not the problem there, I just had very ingrained search habits. I would instead also move in the more qualitative direction here, so asking asking people what features they like perhaps in comparison to how they normally search for images or what features are lacking that would prevent them from using it more often. You won't get a concrete number out of this but hopefully you'll get a sense of where the new search is matching expectations vs. still lacking. You could always take a random sample (say 100) responses and hand-code them into basic categories -- e.g., positive, ambivalent, negative -- to get a more quantitative measure if you'd like.

So the 2 remaining downers are the ?quicksearch= url requirement for the survey to show up here, and the problem of it continuing to show up in that case (even if already completed).

I would caution against this if it's at all avoidable. We hear back from the community pretty quickly if they can't get the survey to disappear (example). It makes people think they didn't answer the survey already (which can be really confusing and frustrating) and could easily lead to bad data if people are just answering to make it disappear.

Without it being the default for some users, most of the survey answers will be expert Common users who know where the link to Media Search is and actively test it. If that's true, couldn't we just put up the full survey on the project page to target the same people? Or put a link to the survey near where people access Media Search?

I'm in favor of this. It sounds like a representative survey is going to be very hard to run without a lot of engineering work. Rather than try to force it into happening, I'd embrace that it won't be representative and make the most of it -- i.e.:

  • Move from questions that will give you strictly quantitative data to more open-ended questions with free-text options like I suggested above.
  • This addresses the challenge that non-expert users frankly probably won't remember what old search looked like well enough to really compare. If you have more control over how people encounter the survey, you can always include a link to the old version so they can test that out to compare.
  • You won't have to hack the survey together in a way that could potentially annoy users and make it harder to intepret the data / run future surveys.

How long are we thinking of leaving this up and how many responses will feel "significant"?

I've only run surveys on Wikipedia where traffic is presumably much higher but also I assume people don't see the survey quite frequently and we see response rates below 1%. WMDE ran an interesting variant of QuickSurveys recently for feedback on proposed changes to templates and might have some insights as well: overview here and T253112 is a good task to start in for seeing what they tried etc.

Quoted Text I'm also worried that many users won't remember what the old version is.

Perhaps we can say "Which search experience do you prefer?" with the options "This version (Special:MediaSearch)", "The default version (Special:Search)", "No preference", and "Other" with an input box.

people are not good at predicting what their behavior will be

Makes sense. After talking with @Ramsey-WMF and @nettrom_WMF, it seems like we should hold off on this question, and perhaps do something more qualitative/open-ended after we have the results of the first survey.

We hear back from the community pretty quickly if they can't get the survey to disappear

@matthiasmullie, you mentioned there's a hack to avoid this - seems like we should go that route. What are the down sides?

How long are we thinking of leaving this up and how many responses will feel "significant"?

Thanks for the examples from WMDE, @Isaac. There's nothing really concrete in there about how many responses is enough, but it seems like they didn't get many. Perhaps we need to self-define something rather small - 25? 50? 100? Whatever we can get in 2 weeks? A month?

@matthiasmullie, you mentioned there's a hack to avoid this - seems like we should go that route. What are the down sides?

None - it's just nasty code, but given how short-lived this is going to be and removed immediately after, that's acceptable.

Thanks for your help @Isaac, it is super helpful!

I would propose we start with the survey in the project page and maybe even a link to it in the Commons Village Pump when the time is right. If we don't get enough results there, then we move to injecting it in the results grid as well. This will allow us to ask both questions at the same time and have space for input fields. The bad news is that it looks like quick survey doesn't allow for open input fields. We could easily get this going with Google Forms as well, and it seems like there is a precedent for that if we get the right privacy policy in place from legal.

Isaac added a comment.Sep 25 2020, 5:03 PM

The bad news is that it looks like quick survey doesn't allow for open input fields. We could easily get this going with Google Forms as well, and it seems like there is a precedent for that if we get the right privacy policy in place from legal.

The freeformTextLabel parameter in the config should allow for an open input field. We did this for surveys on gender where we wanted to give people the option to self-identify. Google Forms is an option though I suspect it reduces response rate and requires a privacy policy as you say (example).

I would propose we start with the survey in the project page and maybe even a link to it in the Commons Village Pump when the time is right. If we don't get enough results there, then we move to injecting it in the results grid as well.

That makes the most sense to me as well.

Perhaps we can say "Which search experience do you prefer?" with the options "This version (Special:MediaSearch)", "The default version (Special:Search)", "No preference", and "Other" with an input box.

@CBogen yeah, I think that's a step in the right direction. Essentially making it as easy as possible for them to quickly check what the old one looked like. The one benefit of moving to an external survey is that screenshots could be included as a reminder but I'm not sure that the additional work of an outside survey is worth it unless there are other reasons for it. You might informally poll a few people outside of the project to see if they know what you mean with whatever label you end up using.

Yeah, 1 freeform textarea is possible (along with 1 radio/checkbox question) - it then looks like this: https://www.mediawiki.org/wiki/Extension:QuickSurveys#/media/File:Freeform_text_in_QuickSurveys.png

matthiasmullie removed matthiasmullie as the assignee of this task.Oct 14 2020, 1:31 PM
matthiasmullie moved this task from To Do to Blocked on the Structured Data Engineering board.

Moving to blocked until someone decides we can move forward with this (and how it ought to be configured)

Resurrection time!

Once we've done a bit more tweaking/balancing with the search results, let's do the following:

  • QuickSurvey on the MediaSearch project page
  • Configure the survey with the question: "Which search experience do you prefer?" with the following options:
    • "The new version (Special:MediaSearch)"
    • "The default/old version (Special:Search)"
    • "No preference"
    • "Other" with an input box.

Change 640934 had a related patch set uploaded (by Matthias Mullie; owner: Matthias Mullie):
[operations/mediawiki-config@master] Add Media Search survey

https://gerrit.wikimedia.org/r/640934

IIUC, we want the survey to be on this page: https://commons.wikimedia.org/wiki/Commons:Structured_data/Media_search
And we want it to look like this: (note: I have to drop The - text was too long for the buttons, and there's not much room for tweaking that...)

Patch is ready - if something is wrong with my understanding above, LMK so I can update the configuration of the survey.
If all is well, please confirm and I will prepare for it to be deployed.

Once deployed, the survey won't be visible until this code is posted on https://commons.wikimedia.org/wiki/Commons:Structured_data/Media_search:

== MediaSearch survey ==
<div id="mediasearch-survey"></div>
This is a bunch of text to explain what the survey is about!
Probably also worth explaining that the survey will only be visible if you've not already filled it out. Once you have, you will no longer see the survey.
<div style="clear: both"></div>

The text - of course - can change. The important part is the 2 <div>s: the first one indicates where to attach the survey; they second makes sure it doesn't run into other sections/footer in case the text is shorter than the survey.

Change 628785 abandoned by Matthias Mullie:
[mediawiki/extensions/WikibaseMediaInfo@master] [WIP] Hacky way to load media search survey

Reason:
Will not need this

https://gerrit.wikimedia.org/r/628785

Change 626338 abandoned by Matthias Mullie:
[mediawiki/extensions/QuickSurveys@master] Allow QuickSurveys on special pages

Reason:
Looks like we won't be needing this after all...

https://gerrit.wikimedia.org/r/626338

CBogen added a subscriber: Keegan.Wed, Dec 2, 2:41 PM

Yes, looks good to me! I'm planning to talk to @Keegan next week after the offsite about when we should go ahead and post it on the project page.

Isaac added a comment.EditedWed, Dec 2, 3:51 PM

Just two quick notes:

  • On the This is a bunch of text to explain what the survey is about!... text: editors often (reasonably) assume that the survey follows them around based on their username when in fact whether it shows up or not is browser-based. You'll probably want to clarify that in that text (e.g., "If you use multiple browsers, you may see the survey multiple times and should just dismiss it if you've already taken it") or you'll likely get some questions along the lines of "I thought I already took this" or "Please make this go away".
  • I don't see a dismiss button for the survey. Apologies for not bringing this up earlier -- I think this is a basic issue with QuickSurveys but I had forgotten. If it's an easy fix, it'd be great to add that. If not, is there a button that could be easily added that is "Prefer not to say" or "Dismiss" or something like that? Otherwise, people who do not want to answer but also don't want to see the survey will get pretty frustrated (in the past, I've shared how to update their local browser cache to remove the survey but this is obviously far from ideal): https://meta.wikimedia.org/wiki/Research:Surveys_on_the_gender_of_editors/Report#QuickSurveys