We'd like to run an A/B test that will have a control group and another group of wikipedia.org portal visitors to be displayed a listing of all languages available, sorted by most articles to least by language, in a easy to use and read dropdown. We want to be sure that the upcoming test is valid/complete from an Analysis standpoint.
The details on the actual wording for this test is in this story's epic: https://phabricator.wikimedia.org/T131526
Bucket testing logic generally is as follows:
- 1 in 200 people are included in EventLogging
- Of those 1 in 200 people, 1 in 10 are included in the test
- Of those 1 in 10 people, 50% go in a test group, with the cohort "lang_dropdown-b", and 50% will go in a control group, with the cohort name "lang_dropdown-a"
- The other chunk of the 200 people gets a NULL (the string null, or the MySQL null, we can detect either).
Note: This test is not meant to decrease the click-through rate of the language by article count links, but to put the sometimes confusing information into a easier to understand interface. This test is also to see if the bounce rate of the portal page can be decreased with this new display of languages, as shown on the portal dashboard: http://discovery.wmflabs.org/portal/#action_breakdown