As part of Article Guidance (T396029) first intervention, we plan to help junior editors when they try to create an article. The intervention is an experiment we want to support using A/B testing.

This ticket covers the experiment plan and preparations. Based on the proposed workflow (T414409) and the metrics defined (T414503), we will design the experiment and complete all preparations so it is ready to launch as soon as the feature is ready. The goal is to prepare the experiment in advance of the product implementation to avoid unnecessary delays.

Hypothesis

If we prepare an A/B test experiment, including how to expose the new feature (entry points) and how to measure the impact (metrics), we'll be ready to start learning about the impact of Article Guidance on the survival rate of articles created by junior editors on mobile.

Decision to be made

Are all of the following true before we start the experiment?

1. The guided article creation flow is instrumented and the instrumentation has been QA'd
2. The experiment is configured in Test Kitchen and has passed security review
3. The outlines have been reviewed and approved by pilot wiki communities
4. The entry points are defined and technically implemented
5. The measurement plan has been signed off by all stakeholders
6. Sample size and statistical power have been calculated
7. The edit tag for tracking articles created via the intervention is implemented
8. The workflow is localised into all pilot wiki languages

Deployment timing

Critical path

Two parallel tracks converge at deploy on test wiki:

Track 1 - Instrumentation:
Instrumentation spec (W.12) -> Security review (W.13) -> Instrumentation implementation and QA (W.13) -> Deploy on test wiki (W.14)

Track 2 - Feature code (already in progress):
Feature implementation (W.13) -> Experiment config and schema (W.14) -> Deploy on test wiki (W.14)

From W.14 onwards:
Deploy on test wiki (W.14) -> Small-scale test (W.15) -> Final review and sign-off (W.15) -> Community outline translation (W.16–W.18) -> Launch (W.18)

⚠️ Note: the critical path now runs through community outline translation, which cannot begin until the English patch demo feedback window closes on April 15 (W.16) and then requires approximately 15 days for communities to translate outlines. Any delay in the patch demo feedback cycle or translation will push the launch date beyond W.18. The full team review scheduled for W.15 will be a key checkpoint to assess whether the launch date is still achievable.

Dependencies

Quit criteria

We would consider stopping work on guided article creation entirely if the Q4 experiment shows both of the following:

1. No statistically significant improvement in 30-day survival rate in the treatment group compared to the control group, across all pilot wikis
2. A meaningful increase in abandonment rate, indicating the workflow is acting as a barrier rather than an enabler

If only one of these is true, we iterate rather than stop. If both are true, the evidence suggests the approach is neither helping editors succeed nor encouraging them to try, and we would reassess the direction entirely.

Q4 Experiment Decision Matrix

Related tasks

• T396029 - Article Guidance epic
• T414503 - Measurements
• T416686 - Entry point analysis
• T414448 - Community collaboration
• T417046 - Instrumentation
• T417200 - Deploy on test wiki
• T417249 - Collaborate with pilot wikis to define guidance
• T416986 - Wikidata labels check
• T418155 - Edit tag
• T418724 - Security review
• T414536 - Experiment design