Page MenuHomePhabricator

Search Relevance test #5: are users happy with the search results they got?
Open, MediumPublic

Description

For the fifth running of the search relevance test, we'd like to ask our users directly if they liked the results they got from their query:

Backend:

  • Do not show the same user (that did the same query) the same survey, ever. 
  • Timeout of the survey will be 60 seconds for survey to display
  • Be sure to only show survey on SERP's (no where else)
  • collect referrer information and join with satisfaction data
  • collect mobile vs desktop

Frontend:

  • Update wording for the question to ask
    • "Are you satisfied with the results you received on this page?"
  • Update wording for answer text
    • Yes, I don't know, No
  • Add link to wiki page where the search relevance test is explained (and offer the talk page for discussion)
    • "Why we are asking this?"
  • Add opt-out for logged in users that would last forever
    • "Opt out of search relevance questions"
  • Add 'close' button in top right of survey display
  • Keep privacy policy link with slight update to wording
    • "Privacy policy information"

To do:

  • write wiki page for why we're testing and request feedback
  • turn on test
  • turn off test
  • analyize results
  • plan next test

Event Timeline

debt created this task.Oct 11 2017, 9:12 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 11 2017, 9:12 PM

If someone clicks the "why are we asking this?" link it might be worth stopping/extending the timeout before the popup disappears so that the question is still there when they've read the answer.

debt added a comment.Oct 12 2017, 8:51 PM

If someone clicks the "why are we asking this?" link it might be worth stopping/extending the timeout before the popup disappears so that the question is still there when they've read the answer.

That makes sense, thanks!

@EBernhardson can you add this functionality into your task list, please? :)

I've put up a patch to core that handles this in a slightly different way. Basically the timeouts will only consider time the page is visible, rather than wall time (as in, the time on the clock on your wall). Then the only thing that needs to happen is resetting the timeout back to 60s when the user clicks the link and goes to a different page to read the privacy statement or why we are running the test.

debt added a comment.Oct 17 2017, 5:20 PM

This test can be run at the same time as other tests, as long as we get bucketing separated.

debt added a comment.Mar 22 2018, 5:18 PM

Keeping this ticket here for future tests - all good practices - even if we don't end up running this exact test.