Page MenuHomePhabricator
Create Task
Maniphest T127878

Write AB test measuring effect of swapping the second and third search result links
Closed, ResolvedPublic
Actions

Description

We know from our Satisfaction schema that users are much more likely to click the second result than the third, but we don't know if it is because the second result is actually better or if people gravitate to the top results. A fairly simple AB test swapping the second and third results should be able to shed some light on how important the ranking of our top 3 search results are.

Details

ProjectBranchLines +/-Subject
mediawiki/extensions/WikimediaEventswmf/1.27.0-wmf.22+22 -52Revert "Test to measure clickthrough position after swapping results"
mediawiki/extensions/WikimediaEventswmf/1.27.0-wmf.21+22 -52Revert "Test to measure clickthrough position after swapping results"
mediawiki/extensions/WikimediaEventsmaster+52 -22Test to measure clickthrough position after swapping results
mediawiki/extensions/CirrusSearchmaster+26 -2Backend support for test swapping second and third result
Customize query in gerrit

Event Timeline

EBernhardson created this task.Feb 23 2016, 9:12 PM
Restricted Application added a project: Discovery. · View Herald TranscriptFeb 23 2016, 9:12 PM
Restricted Application added subscribers: StudiesWorld, Aklapper. · View Herald Transcript
Deskana triaged this task as Low priority.Feb 23 2016, 11:21 PM
Deskana added a project: Discovery-Search (Current work).
Deskana moved this task from Needs triage to On Sprint Board on the Discovery board.Feb 24 2016, 7:10 PM
EBernhardson renamed this task from Run AB test measuring effect of swapping the second and third search result links to Write AB test measuring effect of swapping the second and third search result links.Mar 23 2016, 1:54 AM
EBernhardson moved this task from Incoming to Needs review on the Discovery-Search (Current work) board.
gerritbot added a subscriber: gerritbot.Mar 23 2016, 1:55 AM

Change 279083 had a related patch set uploaded (by EBernhardson):
Backend support for test swapping second and third result

https://gerrit.wikimedia.org/r/279083

gerritbot added a project: Patch-For-Review.Mar 23 2016, 1:55 AM
gerritbot added a comment.Mar 23 2016, 2:01 AM

Change 279084 had a related patch set uploaded (by EBernhardson):
Test to measure clickthrough position after swapping results

https://gerrit.wikimedia.org/r/279084

EBernhardson added a subscriber: mpopov.Mar 24 2016, 4:14 PM

@mpopov a few ideas from a meeting i had yesterday with some external search people:

  • We might want to test out swapping 2 & 3, and separately swapping 3 & 4, as an A/B/C test.
  • The aggregated numbers might be relevant, but most interesting will be queries that exist in both the A and B side. As such it might be usefull to use a larger sample size to ensure we have some overlap.
gerritbot added a comment.Mar 25 2016, 11:25 AM

Change 279083 merged by jenkins-bot:
Backend support for test swapping second and third result

https://gerrit.wikimedia.org/r/279083

ReleaseTaggerBot added a project: MW-1.27-release (WMF-deploy-2016-03-29_(1.27.0-wmf.19)).Mar 25 2016, 12:00 PM
Deskana moved this task from Uncategorised to Scoring and result ordering on the CirrusSearch board.Mar 28 2016, 11:52 PM
gerritbot added a comment.Mar 30 2016, 12:47 PM

Change 279084 merged by jenkins-bot:
Test to measure clickthrough position after swapping results

https://gerrit.wikimedia.org/r/279084

dcausse moved this task from Needs review to Needs Reporting on the Discovery-Search (Current work) board.Mar 30 2016, 2:03 PM
dcausse assigned this task to EBernhardson.Mar 30 2016, 2:29 PM
JustinOrmont added a subscriber: JustinOrmont.Mar 31 2016, 8:51 AM
ReleaseTaggerBot added a project: MW-1.27-release (WMF-deploy-2016-04-05_(1.27.0-wmf.20)).Mar 31 2016, 9:02 AM
Deskana added a subscriber: Deskana.Mar 31 2016, 7:49 PM

Did we file follow-up tasks for this for the analysis and such?

Deskana added a comment.Apr 12 2016, 8:57 PM
In T127878#2165873, @Deskana wrote:

Did we file follow-up tasks for this for the analysis and such?

@EBernhardson This question is still outstanding. When you get a chance, could you answer it? Thanks!

Restricted Application added a project: Discovery-Search. · View Herald TranscriptApr 12 2016, 8:57 PM
EBernhardson added a comment.Apr 12 2016, 9:39 PM

No follow ups have been created afaik

EBernhardson added a comment.Apr 12 2016, 9:40 PM

Looks like this will be ready to disable on thursday, april 14

Deskana removed a project: Discovery-Search.Apr 12 2016, 9:55 PM
gerritbot added a comment.Apr 26 2016, 10:54 PM

Change 285542 had a related patch set uploaded (by EBernhardson):
Revert "Test to measure clickthrough position after swapping results"

https://gerrit.wikimedia.org/r/285542

gerritbot added a comment.Apr 26 2016, 10:54 PM

Change 285543 had a related patch set uploaded (by EBernhardson):
Revert "Test to measure clickthrough position after swapping results"

https://gerrit.wikimedia.org/r/285543

gerritbot added a comment.Apr 26 2016, 11:12 PM

Change 285542 merged by jenkins-bot:
Revert "Test to measure clickthrough position after swapping results"

https://gerrit.wikimedia.org/r/285542

gerritbot added a comment.Apr 26 2016, 11:12 PM

Change 285543 merged by jenkins-bot:
Revert "Test to measure clickthrough position after swapping results"

https://gerrit.wikimedia.org/r/285543

ReleaseTaggerBot added projects: MW-1.27-release (WMF-deploy-2016-04-26_(1.27.0-wmf.22)), MW-1.27-release (WMF-deploy-2016-04-12_(1.27.0-wmf.21)).Apr 27 2016, 12:00 AM
Deskana closed this task as Resolved.May 11 2016, 10:41 PM
Content licensed under Creative Commons Attribution-ShareAlike 3.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL