Overview

The Editing team is planning a controlled experiment to evaluate the impact of the Suggestion Mode MVP. In this experiment, users in the treatment group will be made aware that actionable suggestions exist for specific changes after opening the visual editor (desktop and mobile).

Note: Per T398369, the team is still deciding whether these suggestions will be shown to newcomers immediately or whether they will need to take an explicit action (e.g., tap a button) to appear.

This task is to investigate if enough suggestions will be shown to editors in the treatment group to achieve a sufficient sample size for the analysis.

Requirements

• Establish Baseline Volume: Identify the daily/weekly volume of editors who open the visual editor on desktop and mobile.
• Estimate the percentage of those editing sessions where at least one suggestion is actually available to be shown.
• Consider impacts from different exposure scenarios (i.e immediate display vs explicit action)
• Power Analysis: Determine the minimum sample size required to detect a statistically significant impact on the primary metric (e.g. constructive edit rate)