Page MenuHomePhabricator

A/B Test (Caterpiller): Specifications on the gathering of data for the languages by article count test
Closed, ResolvedPublic

Description

We'd like to run an A/B test that will have a control group and another group of wikipedia.org portal visitors to be displayed a listing of all languages available, sorted by most articles to least by language, in a easy to use and read dropdown. We want to be sure that the upcoming test is valid/complete from an Analysis standpoint.

The details on the actual wording for this test is in this story's epic: https://phabricator.wikimedia.org/T131526

Bucket testing logic generally is as follows:

  • 1 in 200 people are included in EventLogging
  • Of those 1 in 200 people, 1 in 10 are included in the test
  • Of those 1 in 10 people, 50% go in a test group, with the cohort "lang_dropdown-b", and 50% will go in a control group, with the cohort name "lang_dropdown-a"
  • The other chunk of the 200 people gets a NULL (the string null, or the MySQL null, we can detect either).

Note: This test is not meant to decrease the click-through rate of the language by article count links, but to put the sometimes confusing information into a easier to understand interface. This test is also to see if the bounce rate of the portal page can be decreased with this new display of languages, as shown on the portal dashboard: http://discovery.wmflabs.org/portal/#action_breakdown

Event Timeline

debt triaged this task as Medium priority.May 4 2016, 3:31 PM

Thanks @debt. The current prioritisation looks good to me.

Looks good to me. Sounds like we're interested in increasing overall clickthrough / decreasing overall bounce rate with this one. Is that correct, @debt?

Hi @mpopov - that is correct: decrease (overall) bounce and increase clickthough is our goal with this and all tests. :)

Deskana assigned this task to mpopov.
Deskana moved this task from Backlog to Done on the Discovery-Analysis (Current work) board.

Given the above discussion, I think we can consider this one resolved.