Here is the revised Search Move AB Test report and findings for final review.

@mpopov - Assigning over to you to review if you have some time. There's some additional formatting I'd like to do before publishing but the report includes the completed analysis and results. Let me know if you have any questions and thanks in advance for your help!

[ R Markdown source | Analysis Codebase]

Summary of key findings (to be discussed and confirmed with @mpopov):

Search Sessions Initiated

• Data: We reviewed unique search sessions initiated by search location group, reviewing pre and post-deployment data for the 6 test wikis and comparing to similar Wikis not in the test.
• Methods: We fit a linear regression model to infer the impact of the search location move on the number of search sessions initiated and confirm any statistical difference between pre and post-deployment data.
• Results: The search location group with the higher number of search sessions initiated varied for each of the test wikis and, based on the results of the model, we were not able to confirm any impact of the new search location on the number of search sessions initiated across the test wikis.

Search Sessions Completed

• Data: We also calculated the percent of all search sessions in the AB test that included at least 1 click to one of the results returned by test wiki and search location. Data was restricted to only sessions that had more than zero results returned to them.
• Methods: We used Bayesian statistical analysis and confidence intervals to review the number of sessions completed and not completed for each search location group during the AB test. We reviewed the data overall across the test wikis and on a per wikis basis.
• Results:
  • Overall, users that saw the new search location are 1.098 times more likely to click on at least 1 search result in a session compared to users that saw the header in the old location.
  • On a per wiki basis, the results varied. On Persian Wikipedia, users that saw the new header location are about 1.029 times more likely to click on at least one result during a session than users that saw the old header location. For the other 5 wikis in the test, the estimates varied but the 95% credible intervals contain 1 indicating we do not have sufficient evidence to draw definitive conclusions for these.

Thanks @mpopov! I've made the suggested changes. Here is the link to the final report.

@ovasileva - Assigning to you for sign-off. Let me know if you have any questions.

@MNeisler - apologies for the delay. Report is now published on https://www.mediawiki.org/wiki/Reading/Web/Desktop_Improvements/Features/Search/AB_Test_on_New_Search_Location with a link available to the full version above. Thank you for the detailed report and all the work here!