Page MenuHomePhabricator

Analyze results of A/B test for new search location
Open, MediumPublic



In T261648: Perform A/B test for search location change, we performed an A/B test of the new location of the search bar in hopes that we will create a better search experience for our users. This task related to the success of this experiment

Analysis Criteria

  • Which group has a higher rate of search sessions initiated? How does this differ per wiki? How does this differ per levels of activity of the editors (nice to have).
  • Which group has a higher rate of search sessions completed? How does this differ per wiki?
  • Have any other interesting trends emerged?
  • (for logged-out users) Is there any perceived changes in search behavior before/after the change?

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptNov 2 2020, 12:22 PM
ovasileva triaged this task as Medium priority.Nov 2 2020, 12:22 PM
MNeisler added a subscriber: mpopov.

@mpopov Here is the first draft of the Search AB Test analysis report for review.

The report currently includes my initial analysis for search sessions initiated and search sessions completed using some of the methodologies we discussed.
Let me know if you have any questions and thanks in advance for taking the time to review!

I'm also still working on the following changes and will provided an updated report with those changes and any needed revisions following break.

  • Summary of key findings
  • Trends for logged-out users
  • Other interesting trends

Megan and I met on Dec 22 to go over initial feedback and discuss a few changes to the analysis.

Quick update - I'm currently revising the report based on feedback from @mpopov. I'll post an updated report for final review later this week.