Page MenuHomePhabricator

Wikipedia.org: compare old data sets vs latest from A/B test (June 2016)
Closed, ResolvedPublic10 Estimated Story Points

Description

This ticket is to do a "quick" data comparison of older data collected for A/B tests (or otherwise retrieved) with the latest A/B test that was run for the languages by article count.

We have at least two valid data sets to look at: one data set from November 2015 that was collected before any changes were made to the wikipedia.org portal page; and a data set that was collected after we launched the new search box in March 2016.

I'd like to see the comparison, using the typical measurements that we have now (click-through events, what links were clicked on, inaction on the page, etc) and compare what we saw in our last A/B test to the older data.

The goal of this comparison is to determine what the overall changes are to these metrics to show (via data) that the A/B tests are effective at helping visitors to do what they want/need to do on the portal page.

Event Timeline

debt triaged this task as High priority.Jun 22 2016, 1:59 PM
mpopov set the point value for this task to 10.Jul 26 2016, 5:32 PM

Looks good, thanks! Publish it!

To do:

  • Add link to Rmarkdown source on GitHub
  • Add link to download pre-made PDF
  • Make it not-self-contained HTML for faster viewing (images would be downloaded separately)

New URL (which allows additional reports in the future): http://wikimedia-research.github.io/Discovery-Research-Portal/deployments/

PDF on Commons: https://commons.wikimedia.org/wiki/File:Impact_of_Wikipedia.org_Portal_Changes_-_A_Retrospective_Statistical_Analysis.pdf

Added to the analysis section on portal page: https://www.mediawiki.org/wiki/Wikipedia.org_Portal#Analysis_of_A.2FB_Testing:_2016
and to weekly status update