Page MenuHomePhabricator

Deploy the new location of the search bar to new vector and begin A/B test on test wikis
Closed, ResolvedPublic3 Estimated Story Points

Description

Background

In T249363 we moved the search bar to the top of the page to allow for better discoverability. We would like to deploy and test this change on our early adopter wikis

We would like to do two A/B test on the changes planned to the search widget as a part of the desktop improvements project:

Moving the search bar to the header of the page (target 2.5% overall increase in search sessions initiated, monitor and report on search sessions completed)
Switching the current search widget to the new search widget (target 2.5% overall increase in search sessions initiated, monitor and report on search sessions completed)
Individual test will be performed per wiki for logged-in user only (nice to have: all users)

Additional background in T256100, T249366, T251740, and T257698.

This tasks covers the a/b test for changing the location of the search widget.

Note: Because the navigation-first DOM reorder (T261802) is tied to the VectorIsSearchInHeader feature flag, it will also go into effect.

Acceptance criteria

  • Deploy new location of the search bar to new vector
  • Begin A/B test on all early adopter wikis:
  • euwiki, frwiki, fawiki, hewiki, ptwikiversity, frwiktionary
  • Run test for 2 weeks
  • Disable A/B test

Related Objects

Event Timeline

Restricted Application added subscribers: Huji, Aklapper. · View Herald TranscriptSep 16 2020, 3:10 PM
Jdlrobson updated the task description. (Show Details)
Jdlrobson updated the task description. (Show Details)
Jdlrobson updated the task description. (Show Details)
Jdlrobson added a subscriber: Jdlrobson.
Jdlrobson set the point value for this task to 3.Sep 21 2020, 5:23 PM

Possible risks:

  • Rollback will require ops
  • It's not just search in header - it's also DOM changes
  • We're a little concerned 2 weeks won't be enough data.
nray updated the task description. (Show Details)Sep 21 2020, 5:55 PM

Waiting for Olga but I assume I'll be working on this one

phuedx added a subscriber: phuedx.Sep 25 2020, 2:01 PM

Just a note to say that the current sample size for the SearchSatisfaction instrumentation is 1% of all sessions across all wikis. I've discussed this with @ovasileva and we're fairly confident that no further changes need to be made.

Change 630206 had a related patch set uploaded (by Jdlrobson; owner: Jdlrobson):
[operations/mediawiki-config@master] Enable search in header A/B test for logged in users

https://gerrit.wikimedia.org/r/630206

Change 630207 had a related patch set uploaded (by Jdlrobson; owner: Jdlrobson):
[operations/mediawiki-config@master] Move search in header for anons

https://gerrit.wikimedia.org/r/630207

Change 630206 merged by jenkins-bot:
[operations/mediawiki-config@master] Enable search in header A/B test for logged in users

https://gerrit.wikimedia.org/r/630206

Mentioned in SAL (#wikimedia-operations) [2020-09-28T18:12:08Z] <urbanecm@deploy1001> Synchronized wmf-config/InitialiseSettings.php: c7e08bc2bbff6aead186350726d5c1c137cca052: Enable search in header A/B test for logged in users (T263032) (duration: 00m 58s)

Deploy is scheduled for tomorrow at 11am.

Change 630207 merged by jenkins-bot:
[operations/mediawiki-config@master] Move search in header for anons

https://gerrit.wikimedia.org/r/630207

Change 631225 had a related patch set uploaded (by Jdlrobson; owner: Jdlrobson):
[operations/mediawiki-config@master] Always make the search in the header

https://gerrit.wikimedia.org/r/631225

Mentioned in SAL (#wikimedia-operations) [2020-09-30T18:08:05Z] <catrope@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Move search in header for anons (T263032) (duration: 00m 59s)

Change 631225 merged by jenkins-bot:
[operations/mediawiki-config@master] Always make the search in the header for anons

https://gerrit.wikimedia.org/r/631225

Mentioned in SAL (#wikimedia-operations) [2020-09-30T18:28:54Z] <catrope@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Put search in header for anons on all wikis, not just desktop-improvements wikis (T263032) (duration: 00m 59s)

Jdlrobson reassigned this task from Jdlrobson to ovasileva.Oct 5 2020, 5:19 PM
Jdlrobson updated the task description. (Show Details)

We decided that these search changes will be rolled back if we see a 5% overall decrease in search sessions initiated. @ovasileva - Is that correct?

ovasileva added a comment.EditedOct 5 2020, 6:31 PM

We decided that these search changes will be rolled back if we see a 5% overall decrease in search sessions initiated. @ovasileva - Is that correct?

Correct.

[Edit] Correct with a caveat. We will first have a conversation on whether to rollback or to iterate - iteration of the feature might also be possible

ovasileva closed this task as Resolved.Oct 13 2020, 11:18 AM

Resolving this. Disabling the test will be tracked in T265333: End A/B test for search location