Analyse the results of the Portal search box A/B test on or after 2016-01-21
Closed, ResolvedPublic3 Story Points

Ironholds updated the task description. (Show Details)
Ironholds raised the priority of this task from to Needs Triage.
Ironholds claimed this task.
Ironholds added a subscriber: Ironholds.
Restricted Application added subscribers: StudiesWorld, Aklapper. · View Herald TranscriptDec 15 2015, 7:34 PM
Deskana triaged this task as High priority.Dec 15 2015, 9:07 PM
Deskana added a subscriber: Deskana.
Ironholds set Security to None.
Ironholds edited a custom field.
Ironholds renamed this task from Analyse the results of the Portal search box A/B test on or after 2015-12-15 to Analyse the results of the Portal search box A/B test on or after 2016-01-11.Dec 16 2015, 9:08 PM

The initial A/B test code was buggy, meaning we have to relaunch :(.

The initial A/B test code was buggy, meaning we have to relaunch :(.

Can you please either elaborate in this bug as to what the problem was, or point to where the elaboration is? Thanks!

Sure; the JS at the frontend wasn't actually collecting clickthroughs through the search box for the test group. In other words, everything was fine except for the one sort of data we absolutely couldn't do without :/. Accordingly we'll relaunch (with the bug fixed) on 4 january, possibly (still deciding) as part of an A/B/C test with the next iteration as well.

Moving this into the backlog; yes, technically we're waiting for 4th January, but really this is a backlog item. :-)

Deskana lowered the priority of this task from High to Normal.
Ironholds_backup renamed this task from Analyse the results of the Portal search box A/B test on or after 2016-01-11 to Analyse the results of the Portal search box A/B test on or after 2016-01-21.Jan 14 2016, 1:18 AM
debt added a subscriber: debt.Jan 19 2016, 4:33 PM
mpopov added a subscriber: mpopov.Jan 21 2016, 9:13 PM
  • The bar plot is unnecessary, and a simple table of # of users in each group would do the job better.
  • "Bayesian" not "bayesian"

Looks good otherwise; good job.

The substance of the report was signed off on. Some presentational issues remain to be resolved.

I think you should include screenshots of the status quo, test condition 1, and test condition 2.

debt added a comment.Jan 26 2016, 7:37 PM

Oliver will update the final copy and release it. Then, this ticket can be closed.

debt added a comment.Jan 28 2016, 5:31 PM

Here's a copy of the email @Ironholds sent out about this testing effort:

From: Oliver Keyes <okeyes@wikimedia.org>
Date: Thu, Jan 28, 2016 at 4:44 AM
Subject: [discovery] Portal A/B test results available
To: A public mailing list about Wikimedia Search and Discovery projects <discovery@lists.wikimedia.org>

Hey all,

A couple of weeks ago we ran an A/B test on the Wikipedia portal
(www.wikipedia.org) to test whether a more prominent search box,
optionally combined with additional metadata such as small images in
the search results, would increase the rate at which people clicked
through from the portal to one of our projects.

We are delighted to say that the test showed a 1-5% increase in the
clickthrough rate, where both a prominent search box and metadata is
used. Accordingly, once we've resolved concerns about the design's
non-JavaScript usability, we hope to deploy it for all users.

The report can be seen at
https://commons.wikimedia.org/wiki/File:First_Portal_Test.pdf - please
let me know if you have any questions.

For Discovery Analytics,

Oliver Keyes
Count Logula
Wikimedia Foundation

debt closed this task as Resolved.Jan 28 2016, 5:31 PM