The Structured Data team would love to have this to analyze the A/B tests we are doing for the new Media Search on Commons (eg T254388). We were hoping to run that test from August 24-31 2020. This will be extremely helpful in determining whether we should pursue making Media Search default on Commons and whether we should use it in Visual Editor and other file namespace searches on the Wikipedias.

We are working on improving search results on Commons by including structured data (captions, statements) etc.
It would be extremely useful (essential, actually) to be able to tell the impact of changes, to confirm that we're moving in a good direction.

Adding my support to having this tool available! As I've been working with the Structured Data team to determine what metrics we want to use to measure the impact of upcoming tests, it's become more and more clear to me that what we're doing is generally what the Discovery team were doing a couple of years ago. The hewiki report created by the tool that Mikhail linked in the descriptions contains a lot of the metrics we've been discussing with the SD team (as well as a lot of additional ones). The tool also analyzes interleaved A/B tests, something the team are planning on doing. Having all of that readily available to enable iterating on experiments with streamlined analysis would take a lot of work out of it!

Note: the "quick & simple analysis of preference from interleaved A/B tests" as described at the end of this issue's description should be good enough for the test that we'd like to run next week.
Knowing the preferential set of results is good enough to drive our immediate work.

The detailed analysis would be welcome, but is not as urgent, and I'll defer to @CBogen or @Ramsey-WMF if and when we'd need that level of detail!

I agree with Matthias - the detailed analysis would be very welcome but is not urgent. I suspect with the work on search that we are focused on, especially as we consider moving to the Wikipedias, that this will become even more welcome next quarter (in October or so).

The "quick & simple analysis of preference from interleaved A/B tests" is urgent, however. I'm hoping that @nettrom_WMF and/or @mpopov can help us run this after the A/B test ends on Aug 31.

Agreed!

In T260706#6399933, @CBogen wrote:

I agree with Matthias - the detailed analysis would be very welcome but is not urgent. I suspect with the work on search that we are focused on, especially as we consider moving to the Wikipedias, that this will become even more welcome next quarter (in October or so).

The "quick & simple analysis of preference from interleaved A/B tests" is urgent, however. I'm hoping that @nettrom_WMF and/or @mpopov can help us run this after the A/B test ends on Aug 31.

Adding my support for this tool as well! The web team is planning to run two AB tests this quarter on planned changes to the search widget as part of the desktop improvements project. Many of the metrics provided with this tool would be extremely relevant and helpful when assessing the impact of these changes.

This may get pushed to next quarter because we've prioritized a new task - T266714 - ahead of this one. That other task is geared toward supporting our commitments to Accessible Content Data, and has the benefit of being timely and enabling data exploration post US-election.

Throwing this in here so it's not forgotten: https://github.com/bearloga/interleaved-python

There's even a CausalImpact-inspired report writer function:

from interleaved import Experiment

ex = Experiment(
    queries = data[data['event'] == 'click']['search_id'].to_numpy(),
    clicks = data[data['event'] == 'click']['ranking_function'].to_numpy()
)
ex.bootstrap(seed=42)

print(ex.summary(ranker_labels=['New Algorithm', 'Old Algorithm'], rescale=True))

 In this interleaved search experiment, 906 searches were used to determine whether the
results from ranker 'New Algorithm' or 'Old Algorithm' were preferred by users (based on
their clicks to the results from those rankers interleaved into a single search result
set).

 The preference statistic, as defined by Chapelle et al. (2012), was estimated to be 74.3%
with a 95% (bootstrapped) confidence interval of (70.0%, 77.9%) on [-100%, 100%] scale
with -100% indicating total preference for 'Old Algorithm', 100% indicating total
preference for 'New Algorithm', and 0% indicating complete lack of preference between the
two -- indicating that the users had preference for ranker 'New Algorithm'.