Page MenuHomePhabricator

SameAs A/B test preliminary analysis
Closed, ResolvedPublic

Description

Background

We would like to take a look at the sameAs A/B test data to ensure no major negative changes have occurred since the A/B test was rolled out

Acceptance criteria

  • Answer the following questions:
  • Are pageviews in the test and control groups dramatically different?
  • Are there any other reasons for concern?
  • Generate a graph of the following:
  • pageviews per day in test and control groups for enwiki
  • pageviews per day in test and control groups for all wikis except enwiki
  • pageviews per day in test and control groups for all wikis

Event Timeline

ovasileva triaged this task as High priority.Dec 5 2018, 11:03 AM
ovasileva created this task.
mpopov moved this task from Triage to Doing on the Product-Analytics board.Dec 6 2018, 9:19 PM
mpopov added a comment.Dec 7 2018, 5:34 PM

Things are looking okay so far:

Would have been nice to see a visible positive change soon after deployment so that's a little disappointing but at least there doesn't seem to be anything bad going on. Will follow up in January in T211191

(Code & queries over at https://github.com/wikimedia-research/SEO-Experiment-SameAsProp)

mpopov closed this task as Resolved.Dec 10 2018, 2:42 PM

@ovasileva feel free to re-open if you have additional questions