Page MenuHomePhabricator

Write AB test measuring effect of swapping the second and third search result links
Closed, ResolvedPublic

Description

We know from our Satisfaction schema that users are much more likely to click the second result than the third, but we don't know if it is because the second result is actually better or if people gravitate to the top results. A fairly simple AB test swapping the second and third results should be able to shed some light on how important the ranking of our top 3 search results are.

Details

Related Gerrit Patches:
mediawiki/extensions/WikimediaEvents : wmf/1.27.0-wmf.22Revert "Test to measure clickthrough position after swapping results"
mediawiki/extensions/WikimediaEvents : wmf/1.27.0-wmf.21Revert "Test to measure clickthrough position after swapping results"
mediawiki/extensions/WikimediaEvents : masterTest to measure clickthrough position after swapping results
mediawiki/extensions/CirrusSearch : masterBackend support for test swapping second and third result

Event Timeline

Restricted Application added a project: Discovery. · View Herald TranscriptFeb 23 2016, 9:12 PM
Restricted Application added subscribers: StudiesWorld, Aklapper. · View Herald Transcript
Deskana triaged this task as Low priority.Feb 23 2016, 11:21 PM
EBernhardson renamed this task from Run AB test measuring effect of swapping the second and third search result links to Write AB test measuring effect of swapping the second and third search result links.Mar 23 2016, 1:54 AM

Change 279083 had a related patch set uploaded (by EBernhardson):
Backend support for test swapping second and third result

https://gerrit.wikimedia.org/r/279083

Change 279084 had a related patch set uploaded (by EBernhardson):
Test to measure clickthrough position after swapping results

https://gerrit.wikimedia.org/r/279084

@mpopov a few ideas from a meeting i had yesterday with some external search people:

  • We might want to test out swapping 2 & 3, and separately swapping 3 & 4, as an A/B/C test.
  • The aggregated numbers might be relevant, but most interesting will be queries that exist in both the A and B side. As such it might be usefull to use a larger sample size to ensure we have some overlap.

Change 279083 merged by jenkins-bot:
Backend support for test swapping second and third result

https://gerrit.wikimedia.org/r/279083

Change 279084 merged by jenkins-bot:
Test to measure clickthrough position after swapping results

https://gerrit.wikimedia.org/r/279084

Did we file follow-up tasks for this for the analysis and such?

Did we file follow-up tasks for this for the analysis and such?

@EBernhardson This question is still outstanding. When you get a chance, could you answer it? Thanks!

Restricted Application added a project: Discovery-Search. · View Herald TranscriptApr 12 2016, 8:57 PM

No follow ups have been created afaik

Looks like this will be ready to disable on thursday, april 14

Change 285542 had a related patch set uploaded (by EBernhardson):
Revert "Test to measure clickthrough position after swapping results"

https://gerrit.wikimedia.org/r/285542

Change 285543 had a related patch set uploaded (by EBernhardson):
Revert "Test to measure clickthrough position after swapping results"

https://gerrit.wikimedia.org/r/285543

Change 285542 merged by jenkins-bot:
Revert "Test to measure clickthrough position after swapping results"

https://gerrit.wikimedia.org/r/285542

Change 285543 merged by jenkins-bot:
Revert "Test to measure clickthrough position after swapping results"

https://gerrit.wikimedia.org/r/285543

Deskana closed this task as Resolved.May 11 2016, 10:41 PM