Page MenuHomePhabricator

Search Relevance test #5: are users happy with the search results they got?
Closed, InvalidPublic

Description

For the fifth running of the search relevance test, we'd like to ask our users directly if they liked the results they got from their query:

Backend:

  • Do not show the same user (that did the same query) the same survey, ever. 
  • Timeout of the survey will be 60 seconds for survey to display
  • Be sure to only show survey on SERP's (no where else)
  • collect referrer information and join with satisfaction data
  • collect mobile vs desktop

Frontend:

  • Update wording for the question to ask
    • "Are you satisfied with the results you received on this page?"
  • Update wording for answer text
    • Yes, I don't know, No
  • Add link to wiki page where the search relevance test is explained (and offer the talk page for discussion)
    • "Why we are asking this?"
  • Add opt-out for logged in users that would last forever
    • "Opt out of search relevance questions"
  • Add 'close' button in top right of survey display
  • Keep privacy policy link with slight update to wording
    • "Privacy policy information"

To do:

  • write wiki page for why we're testing and request feedback
  • turn on test
  • turn off test
  • analyize results
  • plan next test

search-rel-test_5.png (413×583 px, 55 KB)

Event Timeline

If someone clicks the "why are we asking this?" link it might be worth stopping/extending the timeout before the popup disappears so that the question is still there when they've read the answer.

If someone clicks the "why are we asking this?" link it might be worth stopping/extending the timeout before the popup disappears so that the question is still there when they've read the answer.

That makes sense, thanks!

@EBernhardson can you add this functionality into your task list, please? :)

I've put up a patch to core that handles this in a slightly different way. Basically the timeouts will only consider time the page is visible, rather than wall time (as in, the time on the clock on your wall). Then the only thing that needs to happen is resetting the timeout back to 60s when the user clicks the link and goes to a different page to read the privacy statement or why we are running the test.

This test can be run at the same time as other tests, as long as we get bucketing separated.

Keeping this ticket here for future tests - all good practices - even if we don't end up running this exact test.

CBogen lowered the priority of this task from Medium to Low.Aug 27 2020, 10:05 PM
CBogen subscribed.

Closing because this is now obsolete. If we are able to get back to this work, we can consult these old tickets for reference but we should start over.