EPIC: [Portal A/B/C test 2]: Make search type-ahead better with image and wikidata description on Wikipedia.org
Closed, ResolvedPublic

Description

Build an A/B/C (3-condition) test to test:

  1. The default experience ("test-group-a")
  2. The search box from the first A/B test ("test-group-b")
  3. The search box from the first test, plus typeahead improvements ("test-group-c")

The logic for admitting people to the test:

  1. 1 in 200 people get into the overall schema
  2. Of these, 1 in 10 get into the test. The remaining 9/10 get the default experience and are not in a test group - they are for dashboarding
  3. Of the 1 in 10, 1/3rd of users are distributed to each test group.

We should keep the same client technical requirements that the default eventlogging has, unless doing so would damage the quality of the data. In that case, we should keep those same technical requirements for the non-tested (9/10) group. If that's not possible, let me know.

MSyed updated the task description. (Show Details)
MSyed raised the priority of this task from to Needs Triage.
MSyed added projects: Epic, Discovery.
MSyed added subscribers: Aklapper, MSyed.
MSyed set Security to None.
Deskana renamed this task from Make search type-ahead better with image and wikidata description on Wikipedia.org to Portal A/B test: Make search type-ahead better with image and wikidata description on Wikipedia.org.Sep 18 2015, 5:34 PM
Deskana moved this task from Needs triage to UX on the Discovery board.Sep 22 2015, 5:20 PM
Restricted Application added a subscriber: StudiesWorld. · View Herald TranscriptDec 9 2015, 6:47 PM
JGirault renamed this task from Portal A/B test: Make search type-ahead better with image and wikidata description on Wikipedia.org to EPIC: [Portal A/B test 2]: Make search type-ahead better with image and wikidata description on Wikipedia.org.Dec 9 2015, 10:25 PM
debt renamed this task from EPIC: [Portal A/B test 2]: Make search type-ahead better with image and wikidata description on Wikipedia.org to EPIC: [Portal A/B/C test 2]: Make search type-ahead better with image and wikidata description on Wikipedia.org.Jan 20 2016, 12:32 AM
debt triaged this task as Normal priority.Jan 20 2016, 8:13 PM
debt added a subscriber: debt.
debt claimed this task.Jan 26 2016, 8:49 PM

assigning this to me for now - until we can close it completely. all dev tasks are completed.

Here's a copy of the email @Ironholds sent out about this testing effort:

From: Oliver Keyes <okeyes@wikimedia.org>
Date: Thu, Jan 28, 2016 at 4:44 AM
Subject: [discovery] Portal A/B test results available
To: A public mailing list about Wikimedia Search and Discovery projects <discovery@lists.wikimedia.org>

Hey all,

A couple of weeks ago we ran an A/B test on the Wikipedia portal
(www.wikipedia.org) to test whether a more prominent search box,
optionally combined with additional metadata such as small images in
the search results, would increase the rate at which people clicked
through from the portal to one of our projects.

We are delighted to say that the test showed a 1-5% increase in the
clickthrough rate, where both a prominent search box and metadata is
used. Accordingly, once we've resolved concerns about the design's
non-JavaScript usability, we hope to deploy it for all users.

The report can be seen at
https://commons.wikimedia.org/wiki/File:First_Portal_Test.pdf - please
let me know if you have any questions.

For Discovery Analytics,

Oliver Keyes
Count Logula
Wikimedia Foundation

debt closed this task as Resolved.Jan 28 2016, 5:32 PM
debt moved this task from Done to Completed on the Discovery-Portal-Sprint board.Feb 4 2016, 11:14 PM