Page MenuHomePhabricator

Analyze results of A/B test for displaying sister project search results (test #2)
Closed, ResolvedPublic6 Estimated Story Points

Description

Background:

Based on the results from T149806 (see: T156300), this second A/B test for displaying sister project search results in a sidebar will have just one test group (displaying the project results based on recall) and one control group. Since the test results showed that we didn't get enough clickthroughs - mostly based on two bugs (T158935 and T158937) and the fact that the zero results rate for the queries entered on the original 4 wikis was higher than average - we decided to add in 4 additional wikipedias to be tested in this round.

This new test (T160004) is expected to last at least a week and will be run on the following wikipedias:

  • Persian (tested in T149806)
  • Italian (tested in T149806)
  • Catalan (tested in T149806)
  • Polish (tested in T149806)
  • Arabic
  • French
  • German
  • Russian

This ticket is to analyze the results of this second test, based on:

  • if the user clicked on an individual result and what wiki project that result came from
  • what position in the list was the selected result
  • if the user clicked on the 'more from' on any wiki project result that was displayed
  • important to compare control group that has sister wiki results vs test group that also has sister wiki results

Event Timeline

mpopov set the point value for this task to 6.
mpopov moved this task from Backlog to In progress on the Discovery-Analysis (Current work) board.

I'll definitely take a look next week!

Great job @mpopov ! I just sent my comments through PR and email. :-)

2nd draft up now :) unless anyone objects, I'll upload the PDF to Commons on Monday

Added link to the Discovery Analysis team page, thanks @mpopov, @chelsyx and @TJones for all your great work and review help!