Page MenuHomePhabricator

Analyze results of A/B test for new search widget
Open, MediumPublic

Description

Description

In T249297: Deploy the new Vue.js search experience we will deploy an A/B test for the new search widget comparing the old and new versions of the widget. This task is related to analyzing success of this experiment

Analysis Criteria

  • Which group has a higher rate of search sessions initiated? How does this differ per wiki? How does this differ per levels of activity of the editors.
  • Which group has a higher rate of search sessions completed? How does this differ per wiki? Per editor bucket?
  • Have any other interesting trends emerged?
  • (for logged-out users) Is there any perceived changes in search behavior before/after the change?

Related Objects

StatusSubtypeAssignedTask
OpenNone
Resolvedovasileva
ResolvedVolker_E
Resolvedphuedx
Resolved eprodromou
ResolvedEvanProdromou
ResolvedPeter.ovchyn
ResolvedPeter.ovchyn
ResolvedPeter.ovchyn
Resolved eprodromou
ResolvedPeter.ovchyn
Resolveddaniel
Resolveddaniel
Resolveddaniel
Resolvedholger.knust
Resolvedholger.knust
Resolvedholger.knust
OpenNone
OpenNone
OpenNone
ResolvedSpikeovasileva
Resolvedphuedx
ResolvedJdrewniak
Resolvedalexhollender
Resolvedovasileva
Resolved Jdlrobson
ResolvedJdrewniak
ResolvedJdrewniak
ResolvedNiedzielski
ResolvedMNeisler
ResolvedMNeisler
Resolvedovasileva
Resolvedphuedx
Resolvedovasileva
Resolvedovasileva
Resolvednray
OpenNone
OpenNone
OpenNone
OpenMNeisler
Resolvedovasileva
ResolvedEdtadros
DeclinedNone
DeclinedNone
ResolvedJdrewniak
ResolvedJdrewniak
Opennnikkhoui
Resolvednnikkhoui
Resolved Jdlrobson
Duplicateovasileva
ResolvedMNeisler
ResolvedVolker_E
Resolvedphuedx
Resolvedovasileva
Resolvedovasileva
Resolvedsbassett
Resolvedjlinehan
Openjlinehan
ResolvedNone
Resolvedjlinehan
Resolvedjlinehan
ResolvedOttomata
ResolvedOttomata
ResolvedSpike Jdlrobson
Resolvedjlinehan
Openjlinehan
Resolvedjlinehan
Resolvedjlinehan
Resolvedjlinehan
Resolvedjlinehan
Resolved Jdlrobson
Resolvedjlinehan
Resolvedjlinehan
Opendr0ptp4kt
Resolvedcolewhite
DeclinedNone
ResolvedNone
Resolvedjlinehan
OpenNone
OpenNone
Openovasileva
Resolvedphuedx
Openalexhollender
Resolvedovasileva
DuplicateNone
Resolvedalexhollender
Openphuedx

Event Timeline

The AB impact analysis report is still in progress but here is an initial look at the number of search sessions initiated by wiki and search group by users in the AB test.

Data below reflects events logged in SearchSatisfaction from 10 March 2021 through 16 March 2021 for all logged-in users that initiated a search on one of the test wikis. Note: While the AB test was run through 30 March 2021, the last week of data was incomplete due to a regression identified in T274869 and was excluded.

Search Sessions Initiated in Widget AB Test by Wiki

WikiOld Search Widget (Control)New Search Widget (Treatment)Both Groups
bnwiki11088198
dewikivoyage502676
euwiki380426806
fawiki99114252416
frwiki120871141323500
frwiktionary85410361890
hewiki69220352727
kowiki5998421441
ptwiki184520913936
ptwikiversity161531
srwiki203432635
trwiki8558191674
vecwiki22325
All 13 test wikis186842067139355

Overall, the number of search sessions initiated by users that saw the new search widget was 10.6% higher than the number of search sessions initiated by users that saw the old search widget. However, results are variable on a per wiki basis and further analysis is needed to infer the impact of the search widget on these observed numbers.

Search Sessions Initiated in Widget AB Test by User's Edit Count

The new search widget group had more search sessions initiated across all experience levels (based on the user's number of cumulative edits) and there was not a significant variation in the percent difference between the two test groups across the 5 different edit count buckets.

(cc @ovasileva)