Next steps:

1. An engineer will pick up this ticket and coordinate the conversations

Update: We met with Sam about this today.

The mixin implementation is a bit complex and difficult to visualize fully. At the very least, it does involve creating a new instrument in WikimediaEvents and then potentially update some schemas related to adding anything new. Aside from figuring out an accurate session end, it’s challenging to identify other potential limitations of the specification at this stage. It would be great if we could work on a proof of concept based on Sam's proposal to better understand its feasibility. @Jdrewniak I'll defer to you what you think we should do next.

You can also refer to this: https://www.mediawiki.org/wiki/Readers/Web/Metrics_Platform_Adoption/Search_Recommendations_Hypothesis_Testing_2024

The hypothesis we want to prove with this test is:

“Users who interact with this feature in mobile web (Minerva) will have 5% longer sessions than those who do not.”

In order to prove that, I think we should gather the following metrics:

• key data
  • session_id - [unique ID] An identifier that will allow us to split data on a per-session basis
  • session_length - [integer in minutes] The length of the session. Per T378072, this will be incremented on a per-minute basis. Sessions less than 1 minute will be recorded as having 0 length.
  • bucket - [control, test] Whether the user is receiving the test treatment or not.
• user interaction events (values for the 'action' property )
  • empty_search_suggestion_impression - Only for the test treatment, when a user sees the new search suggestions, after clicking into the search box.
  • empty_search_suggestion_click - Only for the test treatment, When a user clicks on a search suggestion.
  • search_suggestion_impression - For both treatments, after a user types into the search box and is presented with search suggestions based on their query.
  • search_suggestion_click - For both treatments, after being presented search suggestions after typing, the user clicks on one.

Those data points should be enough to answer the following questions:

• How often do users interact with the new empty-state suggestions vs the type-ahead search suggestions on a per session basis, as well as in total.
• Do the new empty-state search suggestions have an impact on sessions length vs the type-ahead search suggestion vs no interaction with either feature?

As we move towards a more self-contained approach for data collection, we should probably set up a separate schema for this AB test as well.

@jwang We've come up with some specs for the experiment. Can you review and see if there's anything missing or that we need to add?

Moving to ready for signoff - Olga to review this week, then Jan to schedule meeting early next week with Jennifer to discuss further

@Jdrewniak, thanks for the instrumentation proposal. Does the team plan to instrument the eventlogging in metric platform schema? If so, the platform has designed several fields to capture the info, like action, action_subtype, action_source, action_context.

@ovasileva, can you clarify your measurement questions for this experiment? Is the statement below correct as your major hypothesis?
Users who interact with this feature on the mobile web (Minerva) in the experiment group will have 5% longer session length than those who are not exposed to this feature in the control group.

I can draft an instrumentation spec based on your measurement questions and @Jdrewniak's inputs next week.

@jwang Yes it will be to Metrics Platform

I have drafted the instrumentation spec for empty search recommendation AB test and listed the events that need to be tracked to calculate each metric. Please review and decide whether/how to instrument them. Additionally, in preparing for the legal/safety/security review, I started drafting a measurement plan. Feel free to replace the placeholders with the specifics of our final plan.

In T378103#10322109, @jwang wrote:

@Jdrewniak, thanks for the instrumentation proposal. Does the team plan to instrument the eventlogging in metric platform schema? If so, the platform has designed several fields to capture the info, like action, action_subtype, action_source, action_context.

@ovasileva, can you clarify your measurement questions for this experiment? Is the statement below correct as your major hypothesis?
Users who interact with this feature on the mobile web (Minerva) in the experiment group will have 5% longer session length than those who are not exposed to this feature in the control group.

@jwang - yes, this is the correct hypothesis.

I can draft an instrumentation spec based on your measurement questions and @Jdrewniak's inputs next week.