Page MenuHomePhabricator

Set up A/B test for new search widget
Closed, ResolvedPublic3 Estimated Story Points

Description

Description

We would like to do two A/B test on the changes planned to the search widget as a part of the desktop improvements project:

  • Moving the search bar to the header of the page (target 2.5% overall increase in search sessions initiated, monitor and report on search sessions completed)
  • Switching the current search widget to the new search widget (target 2.5% overall increase in search sessions initiated, monitor and report on search sessions completed)
  • Individual test will be performed per wiki for logged-in user only (nice to have: all users)

Additional background in T256100, T249366, T251740, and T257698.

This tasks covers the a/b test for switching the current search widget with the new widget.

Acceptance criteria

  • Ensure bucketing is available for A/B test of new vs old search widget

Event Timeline

ovasileva triaged this task as Medium priority.Aug 31 2020, 2:43 PM
ovasileva created this task.
Jdlrobson added a subscriber: Jdlrobson.

I assume this is going to be blocked until T261648 has run. We'll want to move the search into the header and close out that A/B test before running this one. There's no point in analysing this task until that happens.

I assume this is going to be blocked until T261648 has run. We'll want to move the search into the header and close out that A/B test before running this one. There's no point in analysing this task until that happens.

👍

Blocked on QA for new fields and A/B test and roll out to office wiki T262207. Earliest date we can deploy is 14th September. Assigning to Olga to decide when this can become unblocked

ovasileva raised the priority of this task from Medium to High.Dec 9 2020, 5:13 PM
ovasileva renamed this task from Perform A/B test for new search widget to Set up A/B test for new search widget.Dec 14 2020, 6:23 PM
ovasileva updated the task description. (Show Details)
ovasileva updated the task description. (Show Details)
ovasileva updated the task description. (Show Details)

@Jdlrobson mentioned the issue that we discovered around the same user's ID being different across wikis and the user therefore potentially receiving different experiences on different wikis. We can resolve with the following:

ServiceWiring.php
$BUCKETS = [
    /* ... */
];

$lookup = CentralIdLookup::factoryNonLocal();
$id = null;
if ( $lookup ) {
  $id = $lookup->centralIdFromLocalUser( $user );
}
// The central ID lookup failed?
if ( !$id ) {
  $id = $user->getId();
}

$bucketIndex = $id % count( $BUCKETS );
ovasileva set the point value for this task to 3.Dec 15 2020, 6:22 PM

Change 656966 had a related patch set uploaded (by Phuedx; owner: Phuedx):
[mediawiki/skins/Vector@master] WIP: Add A/B test harness for Core/Vue.js-based search widgets

https://gerrit.wikimedia.org/r/656966

Change 657085 had a related patch set uploaded (by Phuedx; owner: Phuedx):
[mediawiki/skins/Vector@master] Open up search widget treatment A/B test

https://gerrit.wikimedia.org/r/657085

https://gerrit.wikimedia.org/r/656966 adds the A/B test, which is enabled or disabled using the VectorSearchTreatmentABTest config variable. https://gerrit.wikimedia.org/r/657085, a follow-on change, opens up the A/B test to anons without additional delay or significant complexity.

@ovasileva: You've said (and the task's description also says) that this is a nice to have. That second patch demonstrates that this is possible, trivially so. However, we have to change the unit of variance – I think this is the correct term – to the user's session ID. @MNeisler: Would this complicate your analysis?

https://gerrit.wikimedia.org/r/656966 adds the A/B test, which is enabled or disabled using the VectorSearchTreatmentABTest config variable. https://gerrit.wikimedia.org/r/657085, a follow-on change, opens up the A/B test to anons without additional delay or significant complexity.

@ovasileva: You've said (and the task's description also says) that this is a nice to have. That second patch demonstrates that this is possible, trivially so. However, we have to change the unit of variance – I think this is the correct term – to the user's session ID. @MNeisler: Would this complicate your analysis?

@phuedx Good news! The change to user's session id shouldn't complicate the analysis to calculate the key metrics identified above (search sessions initiated and completed).

https://gerrit.wikimedia.org/r/656966 adds the A/B test, which is enabled or disabled using the VectorSearchTreatmentABTest config variable. https://gerrit.wikimedia.org/r/657085, a follow-on change, opens up the A/B test to anons without additional delay or significant complexity.

@ovasileva: You've said (and the task's description also says) that this is a nice to have. That second patch demonstrates that this is possible, trivially so. However, we have to change the unit of variance – I think this is the correct term – to the user's session ID. @MNeisler: Would this complicate your analysis?

@phuedx Good news! The change to user's session id shouldn't complicate the analysis to calculate the key metrics identified above (search sessions initiated and completed).

Great news!

I've removed the -2 from the second patch. Thanks, @MNeisler!

Change 657085 abandoned by Phuedx:
[mediawiki/skins/Vector@master] Open up search widget treatment A/B test

Reason:

https://gerrit.wikimedia.org/r/657085

Change 656966 merged by jenkins-bot:
[mediawiki/skins/Vector@master] Add search widget treatment A/B test

https://gerrit.wikimedia.org/r/656966

Edtadros added a subscriber: Edtadros.

@phuedx test steps please.

The following are the QA steps that I went through with @Edtadros yesterday:

  1. Add the following to your LocalSettings.php:
php,name=/path/to/mediawiki/LocalSettings.php
$wgVectorSearchTreatmentABTest = true;
  1. Login to your dev wiki as Admin
  2. Observe that:
    1. The magnifying glass is on the right-hand-side of the search widget
    2. The body element doesn't have the skin-vector-search-vue class

  1. Login to your dev wiki as a user with an even ID (e.g. Phuedx-test-2 on my dev wiki has an ID of 4)
  2. Observer that:
    1. The magnifying glass is on the left-hand-side of the search widget
    2. The body element has the skin-vector-search-vue class

Per the above. Moving this to Ready for Signoff because this can't be QA'd in production until the WVUI library has been deployed (see T271353: Add WVUI as a vendor library in core).

phuedx removed phuedx as the assignee of this task.Feb 1 2021, 6:03 PM

☝️ Per our discussion yesterday, we'll sign this off now and test the change after we've enabled the A/B test on the Beta Cluster.

☝️ Per our discussion yesterday, we'll sign this off now and test the change after we've enabled the A/B test on the Beta Cluster.

Now added QA requirement under T259798: Deploy the new Vue.js search experience to the Beta-Cluster and Test Wikipedia. Resolving this!