Page MenuHomePhabricator

[SPIKE] Decide whether the Suggestion Mode MVP controlled experiment will run on Test Kitchen
Closed, ResolvedPublic

Description

In T404600, the Editing Team will be running a controlled experiment to evaluate the impact of the Suggestion Mode MVP.

This task involves the work of deciding whether this experiment will run on Test Kitchen.

Decision(s) to be made

  • 1. Will the Suggestion Mode MVP controlled experiment (T404600) run on Test Kitchen?

Open questions

  • 1. Does Test Kitchen support the metrics the Suggestion Mode MVP controlled experiment (T404600) specifies?
  • 2. Does Test Kitchen support multi-variate tests?
    • Yes, per offline discussion with @JVanderhoop-WMF and we came to think that a multi-variate test is not needed for this experiment.
  • 3. How (if at all) has the percentage of logged out traffic we can include in a controlled experiment increased from the current 0.1% threshold?

Done

  • Answers to all "Open questions" are documented
  • Paths forward for all "Decision(s) to be made" are documented

Event Timeline

  1. How (if at all) has the percentage of logged out traffic we can include in a controlled experiment increased from the current 0.1% threshold?

To clarify, the 0.1% threshold is set specifically for English Wikipedia. The other wikis have a 10% max traffic threshold. These limits are set for any experiment that uses the edge_unique identifer, which is needed if we want to include logged-out users in the experiment.

See current Test Kitchen documentation and related task to investigate impacts of increasing the enrollment sampling thresholds.

Test Kitchen guidance also currently states

If you need to have trustworthy insights about the effect of your change on logged-in user behavior, we recommend conducting a separate experiment (either in parallel or as a follow-up) specifically on logged-in users, using mw-user as the identifier type.

This will be important to consider in our experiment design if our primary metric is focused on the impact to registered users.

ppelberg added subscribers: DLynch, mpopov.

Per today's offline discussion w/ @MNeisler, @DLynch, @JVanderhoop-WMF, and @mpopov, we decided to run the Suggestion Mode controlled experiment on Test Kitchen.

The main open question have to do with sampling rates which we will address as part of T421189.