Here is the current draft of the Search Widget AB Report : Repo and Notebook

@mpopov - Assigning to you for review to confirm the approach and findings look correct. Thank you in advance! Please let me know if you have any questions or suggested revisions.

@MNeisler: Great job with the report! As I mentioned to you when you first showed it to me, I really like your use of color in the visualizations and in general how you've applied the lessons learned in the Storytelling with Data workshop :)

I left some comments and suggestions for how to improve some charts via ReviewNB. Let me know if you run into any problems responding to the comments, resolving discussions in that tool. Also, you don't have to make revisions based on the code-related suggestions – those are just for educational purpose and future reference.

My main concern is with the entire "search sessions completed" section. In our 1:1 when we discussed the reservations you had with approach 1 I too was apprehensive about including it, and the results in the draft + the hypothesis you had to explain those results convinced me that approach 1 should be omitted. Furthermore, the issue (intentional bug?) you identified with visitPage events only ever being emitted on pages visited from full-text and never from autocomplete (which I also verified) really makes me question the inclusion of approach 2 as well; and, consequently, the inclusion of that section as a whole. To be clear: there is no fault with your methods, but rather the data (which those methods are applied to) is, perhaps, simply too flawed to yield any useful insights. Since the primary KPI for this search sessions initiated, this should be OK, but we should discuss this in our next technical 1:1.

Re-assigned for iteration based on feedback.

Thanks so much for the review @mpopov! I've incorporated the suggested feedback.

@ovasileva - Here is the revised Search Widget AB Report. Please see summary of high-level findings below.

Per discussions with Mikhail, I've decided to remove the Search Sessions Completed section from the report as the data is too flawed and unreliable to draw any useful insights. We've discussed some of the reasons in our 1:1s (and as described in T275200#7152785 ) but I've documented them below for reference.

But please let me know if you have any questions or concerns about that change or any of the other findings in the report.

Summary of key insights and observations
Logged-in users AB Test

• We observed an average 28.9% increase in the search sessions initiated across early adopter wikis in the AB test; however, there is not sufficient evidence to definitively say that the new search widget led to this increase in observed search sessions initiated.
• Results varied on a per wiki basis, with some observed increases and decreases in search sessions initiated between the two test groups.
  • For 7 out of the 12 early adopter wikis, there was an increase in the number of search sessions initiated in the new search widget test group.
  • Most increases ranged from about 12% to 22% but search sessions initiated were more than double on Serbian (+108.12%) and Hebrew Wikipedia (+192.37%).
  • The highest decreases were seen on German WikiVoyage (-52%) and Bengali Wikipedia (-20%); however, both of these wikis had under 100 search sessions recorded for each search widget type during the AB test
• There were no significant differences in search sessions initiated based on the user's edit count bucket indicating that a user's editing experience is not a factor in the likelihood of them starting a search.

Logged-Out Users Pre and Post Deployment Analysis

• We observed an 8% overall increase (13.4% median increase across early adopter wikis) in search sessions initiated by logged-out users on the early adopter wikis following deployment of the new search widget. Note that since a controlled experiment was not conducted on logged-out users, we are unable to conclude that this observed increase was due to the new search widget.
• Results observed on each early adopter wiki varied, ranging from a -21.86% decrease on Persian Wikipedia to a 240.2% increase on Korean Wikipedia.

Search Sessions Completed Errors

During the search sessions completed analysis, we identified a regression where clicks to the search button (by hitting the 'Enter' key or clicking the search icon in the widget) were not recorded for the new search widget (See T274869#7076438). As an alternative, we reviewed two possible approaches: (1) proportion of sessions where event.action = 'visitPage' were recorded and (2) proportion of sessions that included a direct click to a search result. We found errors in both approaches, which are described below:

(1) Direct Clicks to a Search Rendered Result:

This does accurately reflect the impact of the new widget on search session completion rate as it does not account for sessions where users clicked a result by hitting the enter button or the search icon and it's not clear if the change in the search widget. For example, it's possible that the new search widget (which moved the location of the search icon button from the right of the widget to the left) led to more users selecting the search icon instead of clicking a result which may contribute to this decrease.

(2) Visit Page Events:

According to the schema documentation, these events are created after a user clicks a link in the results; however, it looks like this event is only sent after a user is either directed to the Special:Search page after clicking a search rendered result or if a user clicks a search result on the Special:Search page. It does not appear to be logged if a user clicks on one of the autocomplete search rendered results that appear as you start typing in the search widget and are then taken directly to the article.

Codebase

All done, thank you @MNeisler!