| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | • debt | T112172 EPIC: of epics Wikipedia.org Portal UX tests to run | |||
| Resolved | • debt | T112209 EPIC: [Portal A/B/C test 2]: Make search type-ahead better with image and wikidata description on Wikipedia.org | |||
| Resolved | Ironholds | T121566 Analyse the results of the Portal search box A/B test on or after 2016-01-21 |
Event Timeline
Can you please either elaborate in this bug as to what the problem was, or point to where the elaboration is? Thanks!
Sure; the JS at the frontend wasn't actually collecting clickthroughs through the search box for the test group. In other words, everything was fine except for the one sort of data we absolutely couldn't do without :/. Accordingly we'll relaunch (with the bug fixed) on 4 january, possibly (still deciding) as part of an A/B/C test with the next iteration as well.
Moving this into the backlog; yes, technically we're waiting for 4th January, but really this is a backlog item. :-)
Now needs review by @bearloga. https://github.com/wikimedia-research/PortalSearchBoxTest/blob/master/report.pdf
- The bar plot is unnecessary, and a simple table of # of users in each group would do the job better.
- "Bayesian" not "bayesian"
Looks good otherwise; good job.
The substance of the report was signed off on. Some presentational issues remain to be resolved.
I think you should include screenshots of the status quo, test condition 1, and test condition 2.
Here's a copy of the email @Ironholds sent out about this testing effort:
From: Oliver Keyes <okeyes@wikimedia.org>
Date: Thu, Jan 28, 2016 at 4:44 AM
Subject: [discovery] Portal A/B test results available
To: A public mailing list about Wikimedia Search and Discovery projects <discovery@lists.wikimedia.org>
Hey all,
A couple of weeks ago we ran an A/B test on the Wikipedia portal
(www.wikipedia.org) to test whether a more prominent search box,
optionally combined with additional metadata such as small images in
the search results, would increase the rate at which people clicked
through from the portal to one of our projects.
We are delighted to say that the test showed a 1-5% increase in the
clickthrough rate, where both a prominent search box and metadata is
used. Accordingly, once we've resolved concerns about the design's
non-JavaScript usability, we hope to deploy it for all users.
The report can be seen at
https://commons.wikimedia.org/wiki/File:First_Portal_Test.pdf - please
let me know if you have any questions.
For Discovery Analytics,
Oliver Keyes
Count Logula
Wikimedia Foundation