Page MenuHomePhabricator

EPIC: [Portal A/B/C test 2]: Make search type-ahead better with image and wikidata description on Wikipedia.org
Closed, ResolvedPublic

Description

Test3.png (1×1 px, 441 KB)

Build an A/B/C (3-condition) test to test:

  1. The default experience ("test-group-a")
  2. The search box from the first A/B test ("test-group-b")
  3. The search box from the first test, plus typeahead improvements ("test-group-c")

The logic for admitting people to the test:

  1. 1 in 200 people get into the overall schema
  2. Of these, 1 in 10 get into the test. The remaining 9/10 get the default experience and are not in a test group - they are for dashboarding
  3. Of the 1 in 10, 1/3rd of users are distributed to each test group.

We should keep the same client technical requirements that the default eventlogging has, unless doing so would damage the quality of the data. In that case, we should keep those same technical requirements for the non-tested (9/10) group. If that's not possible, let me know.

Event Timeline

MSyed raised the priority of this task from to Needs Triage.
MSyed updated the task description. (Show Details)
MSyed added subscribers: Aklapper, MSyed.
Deskana renamed this task from Make search type-ahead better with image and wikidata description on Wikipedia.org to Portal A/B test: Make search type-ahead better with image and wikidata description on Wikipedia.org.Sep 18 2015, 5:34 PM
JGirault renamed this task from Portal A/B test: Make search type-ahead better with image and wikidata description on Wikipedia.org to EPIC: [Portal A/B test 2]: Make search type-ahead better with image and wikidata description on Wikipedia.org.Dec 9 2015, 10:25 PM
debt renamed this task from EPIC: [Portal A/B test 2]: Make search type-ahead better with image and wikidata description on Wikipedia.org to EPIC: [Portal A/B/C test 2]: Make search type-ahead better with image and wikidata description on Wikipedia.org.Jan 20 2016, 12:32 AM
debt triaged this task as Medium priority.Jan 20 2016, 8:13 PM
debt subscribed.

assigning this to me for now - until we can close it completely. all dev tasks are completed.

Here's a copy of the email @Ironholds sent out about this testing effort:

From: Oliver Keyes <okeyes@wikimedia.org>
Date: Thu, Jan 28, 2016 at 4:44 AM
Subject: [discovery] Portal A/B test results available
To: A public mailing list about Wikimedia Search and Discovery projects <discovery@lists.wikimedia.org>

Hey all,

A couple of weeks ago we ran an A/B test on the Wikipedia portal
(www.wikipedia.org) to test whether a more prominent search box,
optionally combined with additional metadata such as small images in
the search results, would increase the rate at which people clicked
through from the portal to one of our projects.

We are delighted to say that the test showed a 1-5% increase in the
clickthrough rate, where both a prominent search box and metadata is
used. Accordingly, once we've resolved concerns about the design's
non-JavaScript usability, we hope to deploy it for all users.

The report can be seen at
https://commons.wikimedia.org/wiki/File:First_Portal_Test.pdf - please
let me know if you have any questions.

For Discovery Analytics,

Oliver Keyes
Count Logula
Wikimedia Foundation