Page MenuHomePhabricator

Analyze results of A/B test for new search location
Closed, ResolvedPublic

Description

Description

In T261648: Perform A/B test for search location change, we performed an A/B test of the new location of the search bar in hopes that we will create a better search experience for our users. This task related to the success of this experiment

Analysis Criteria

  • Which group has a higher rate of search sessions initiated? How does this differ per wiki? How does this differ per levels of activity of the editors (nice to have).
  • Which group has a higher rate of search sessions completed? How does this differ per wiki?
  • Have any other interesting trends emerged?
  • (for logged-out users) Is there any perceived changes in search behavior before/after the change?

Event Timeline

MNeisler added a subscriber: mpopov.

@mpopov Here is the first draft of the Search AB Test analysis report for review.

The report currently includes my initial analysis for search sessions initiated and search sessions completed using some of the methodologies we discussed.
Let me know if you have any questions and thanks in advance for taking the time to review!

I'm also still working on the following changes and will provided an updated report with those changes and any needed revisions following break.

  • Summary of key findings
  • Trends for logged-out users
  • Other interesting trends

Megan and I met on Dec 22 to go over initial feedback and discuss a few changes to the analysis.

Quick update - I'm currently revising the report based on feedback from @mpopov. I'll post an updated report for final review later this week.

MNeisler moved this task from Doing to Needs Review on the Product-Analytics (Kanban) board.
MNeisler added a subscriber: MNeisler.

Here is the revised Search Move AB Test report and findings for final review.

@mpopov - Assigning over to you to review if you have some time. There's some additional formatting I'd like to do before publishing but the report includes the completed analysis and results. Let me know if you have any questions and thanks in advance for your help!

[ R Markdown source | Analysis Codebase]

Summary of key findings (to be discussed and confirmed with @mpopov):

Search Sessions Initiated

  • Data: We reviewed unique search sessions initiated by search location group, reviewing pre and post-deployment data for the 6 test wikis and comparing to similar Wikis not in the test.
  • Methods: We fit a linear regression model to infer the impact of the search location move on the number of search sessions initiated and confirm any statistical difference between pre and post-deployment data.
  • Results: The search location group with the higher number of search sessions initiated varied for each of the test wikis and, based on the results of the model, we were not able to confirm any impact of the new search location on the number of search sessions initiated across the test wikis.

Search Sessions Completed

  • Data: We also calculated the percent of all search sessions in the AB test that included at least 1 click to one of the results returned by test wiki and search location. Data was restricted to only sessions that had more than zero results returned to them.
  • Methods: We used Bayesian statistical analysis and confidence intervals to review the number of sessions completed and not completed for each search location group during the AB test. We reviewed the data overall across the test wikis and on a per wikis basis.
  • Results:
    • Overall, users that saw the new search location are 1.098 times more likely to click on at least 1 search result in a session compared to users that saw the header in the old location.
    • On a per wiki basis, the results varied. On Persian Wikipedia, users that saw the new header location are about 1.029 times more likely to click on at least one result during a session than users that saw the old header location. For the other 5 wikis in the test, the estimates varied but the 95% credible intervals contain 1 indicating we do not have sufficient evidence to draw definitive conclusions for these.

@mpopov

I made some final formatting adjustments and added some additional text to clarify findings. See revised report and R Markdown source for review. Please let me know if you have any questions or suggested revisions. Thank you!!

Looks great, @MNeisler! Excellent work.

I would change

While the estimate of the effect of the new search location is 2.4% increase in search sessions initiated, the 95% confidence interval contains 1, meaning we do not have sufficient evidence to draw definitive conclusions.

to

While the estimate of the effect of the new search location is 2.4% increase in search sessions initiated, the 95% confidence interval is [-13.8%, 18.7%], meaning we do not have sufficient evidence to draw definitive conclusions.

Note: Since the 95% CI for exp(coefficient) is [0.862, 1.187] I subtracted 0.862 from 1 to get the lower bound on the increase/decrease scale.

P.S. I recommend updating the citation for the old "wmf" package at the bottom to

Mikhail Popov and Os Keyes (2021). wmfdata: R Tools For Wikimedia Foundation's Analysts And Data Scientists. R package version 0.9.1. https://github.com/wikimedia/wmfdata-r

MNeisler moved this task from Needs Review to Needs Sign-off on the Product-Analytics (Kanban) board.

Thanks @mpopov! I've made the suggested changes. Here is the link to the final report.

@ovasileva - Assigning to you for sign-off. Let me know if you have any questions.

@MNeisler: just a heads-up that the link to the final report is broken :(

@mpopov Thanks for the heads up! I've fixed the link and replaced any references I could find to the old link with the updated link to the final report.

@MNeisler - apologies for the delay. Report is now published on https://www.mediawiki.org/wiki/Reading/Web/Desktop_Improvements/Features/Search/AB_Test_on_New_Search_Location with a link available to the full version above. Thank you for the detailed report and all the work here!