[EPIC] Wikipedia.org A/B Test (Caterpiller): languages by article count
Closed, ResolvedPublic

Description

As a visitor to the wikipedia.org portal page, I want to see a clean interface that allows me to search for a preferred language effectively and efficiently.

As a Product Manager, I want to display a listing of all languages available, sorted by most articles to least by language, in a easy to use and read dropdown. I also want to ensure that a new display treatment of the available languages does not negatively affect the click through rate but will hopefully decrease the amount of bounces from the portal page.

The new format will show a scrolling modal that will be above the fold on the portal page and will list all available languages. When a visitor clicks on the new language dropdown, they will see the familiar look and feel of the existing languages by article count treatment, with the top most languages by article listed first.

The new layout of the page will show a new dropdown that will be centered underneath the search box and:

  • an icon denoting languages will display to the left of the visible text of the dropdown
  • text will display in the visible area of the dropdown and the copy will read: "200 more languages" (or similar icon denoting that more languages are available in the dropdown)
  • the dropdown will include a down arrow that denotes that it is a dropdown in which the users can click on
  • when the user clicks on the dropdown, a modal will appear that will contain all available languages
  • the languages will be ordered by those languages with the most articles listed first to those languages that have the least amount of articles
  • the languages will be segregated into the typical sections, as it currently exists - 1,000,000+ articles, 100,000+ articles, 10,000+ articles and so on
  • the modal will display the book icon to the left hand side of the article amount number

Note: We will use the recently completed code for the re-sorting of the languages, based on the detection of the primary links code

Note: This test is not meant to decrease the click-through rate of the language by article count links, but to put the sometimes confusing information into a easier to understand interface. This test is also to see if the bounce rate of the portal page can be decreased with this new display of languages.

See attached images:

debt created this task.Apr 1 2016, 5:33 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 1 2016, 5:33 PM
debt triaged this task as Normal priority.
debt updated the task description. (Show Details)Apr 7 2016, 5:05 PM
debt renamed this task from [EPIC] Wikipedia.org A/B Test: languages by article count to [EPIC] Wikipedia.org A/B Test (Caterpiller): languages by article count.Apr 26 2016, 4:56 PM
debt claimed this task.
debt added a project: Discovery.
debt updated the task description. (Show Details)Apr 26 2016, 6:55 PM
Jdrewniak updated the task description. (Show Details)May 17 2016, 10:24 AM
debt added a comment.May 17 2016, 1:57 PM

I reached out to Moiz yesterday to see if he has a mock showing how the dropdown will look once it's expanded and how the dropdown will be closed.

Change 292992 had a related patch set uploaded (by Jdrewniak):
T131526 A/B test on wikipedia.org

https://gerrit.wikimedia.org/r/292992

Change 292992 merged by jenkins-bot:
T131526 A/B test on wikipedia.org

https://gerrit.wikimedia.org/r/292992

debt added a comment.Jun 22 2016, 1:26 PM

Analysis is completed - this task is done. Here's the summary of the results of this tests:

One of Discovery’s Wikipedia Portal team’s goals is to make wikipedia.org a more inviting experience. One thing that worried us, from a user experience perspective, was the large section with links to Wikipedia in all the different languages. Specifically, we were worried it might be overwhelming to a new user. To that end, we propose a design change wherein that section is made optional and redesigned to have a cleaner, columnar layout.
We ran A/B test to assess the effect of this change for a week, from 6 June 2016 through 14 June 2016. The test group consisted of 3019 users who received the cleaner design while the control group consisted of 2560 users who received the current design.
We found that the test group had a 3.64% higher probability of interacting with the page and was 1.07 times more likely to engage with the page than the control group. Furthermore, although both groups received the dynamic primary link feature, users in the test group had a 6.82% higher probability (1.09 times more likely) of visiting a Wikipedia in their most preferred language.
Our main goal with this design change was to decrease the bounce rate (increase the engagement rate). We saw a significant improvement in how people engaged with the page when given the cleaner variation, which suggests that we should proceed with this change.

sorted by most articles to least by language

This sounds like "Make it hard to find the only language that I read". Have you considered just alphabetizing instead?

As a thought exercise, imagine that you need to put your mailing address into a web page. Think about how much more difficult it would be to find the name of your country, if the list was sorted by the size of the country.

debt added a comment.Jul 4 2016, 8:35 PM

Hi @Whatamidoing-WMF,

Thanks for your input - we've been kicking around the idea of using an alphabetical language list, rather than the listing by article amounts.

I've created a ticket to run an A/B test on this idea and if it tests well, we'll need an ok by the community to implement it, as I'm not sure how many folks check the portal page to see where their favorite language is in the list by articles.

debt closed this task as Resolved.Jul 18 2016, 10:10 PM

Closing this out now, all testing and analysis is done. More information on this test is here.

debt moved this task from Done to Completed on the Discovery-Portal-Sprint board.Aug 12 2016, 8:00 PM