Analyze results of A/B test for displaying sister project search results (test #2)
Closed, ResolvedPublic6 Story Points

Description

Background:

Based on the results from T149806 (see: T156300), this second A/B test for displaying sister project search results in a sidebar will have just one test group (displaying the project results based on recall) and one control group. Since the test results showed that we didn't get enough clickthroughs - mostly based on two bugs (T158935 and T158937) and the fact that the zero results rate for the queries entered on the original 4 wikis was higher than average - we decided to add in 4 additional wikipedias to be tested in this round.

This new test (T160004) is expected to last at least a week and will be run on the following wikipedias:

  • Persian (tested in T149806)
  • Italian (tested in T149806)
  • Catalan (tested in T149806)
  • Polish (tested in T149806)
  • Arabic
  • French
  • German
  • Russian

This ticket is to analyze the results of this second test, based on:

  • if the user clicked on an individual result and what wiki project that result came from
  • what position in the list was the selected result
  • if the user clicked on the 'more from' on any wiki project result that was displayed
  • important to compare control group that has sister wiki results vs test group that also has sister wiki results
debt created this task.Mar 8 2017, 11:52 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 8 2017, 11:52 PM
mpopov claimed this task.Mar 27 2017, 11:35 PM
mpopov set the point value for this task to 6.
mpopov moved this task from Backlog to In progress on the Discovery-Analysis (Current work) board.
debt added a comment.Mar 28 2017, 11:28 PM

Here's the link to a sample for the second search results for sister projects: https://commons.wikimedia.org/wiki/File:Sister_Projects_search_results_second_test_screen_sample.png

mpopov added subscribers: chelsyx, TJones.

1st draft online: https://wikimedia-research.github.io/Discovery-Search-Test-CrosswikiSidebar-2/

@chelsyx @TJones, would you be able to review sometime next week? (Please and thank you!)

TJones added a comment.Apr 7 2017, 7:32 PM

I'll definitely take a look next week!

Great job @mpopov ! I just sent my comments through PR and email. :-)

Looks great! Notes emailed to @mpopov.

2nd draft up now :) unless anyone objects, I'll upload the PDF to Commons on Monday

debt awarded a token.Apr 24 2017, 3:25 PM
debt closed this task as Resolved.Apr 24 2017, 3:38 PM

Added link to the Discovery Analysis team page, thanks @mpopov, @chelsyx and @TJones for all your great work and review help!