We have defined our research questions, measurements, experiment plan, and leading indicators in the "Add a Link - Measurement Specification". This task contains the experiment plan.
In this A/B test setup, a randomly selected half of users who get the Growth features will get Add a Link tasks, and the other randomly selected half will get unstructured link tasks. We prefer to give users maximum exposure to these tasks and will therefore not give these users any copyedit tasks by default. In other words, for the purposes of this experiment, we’ll change the default difficulty filters from “links” and “copyedit” to just “links”. For wikis that don’t have unstructured link tasks, all those users get “add a link” and in that case we’ll exclude that wiki from the experiment.
We plan to continue to have a Control group that does not get access to the Growth features, which is a randomly selected 20% of new registrations.
Group A: users get “add a link” as their only default task type.
Group B: users get unstructured link task as their only default task type.
Group C: control (no Growth features)
The experiment will run for a limited time, most likely between four to eight weeks. We base this duration on our most recent Variant C/D test, which had a similar setup of randomly assigned groups. That experiment ran for four weeks and the dataset contained 67,598 registrations. Since then, the Growth features have been deployed on several additional wikis, further reducing the need for experiments with a longer duration.
In practice the experiment will start with our four pilot wikis. After two weeks, we will analyze the leading indicators in T277355: Add a link: leading indicators to decide whether to extend the experiment to the rest of the Growth wikis.
Users who registered prior to the experiment and have access to the Growth features will have the unstructured link task replaced by Add a Link. These users will not be part of the experiment itself (because they were not randomly selected into treatment groups), and get the feature so that we can expose it to a larger group of contributors.