INSTRUMENTATION SPECIFICATION: TBD

Testable hypothesis

If we suppress the Welcome notification for new account holders, it will increase constructive activation because the notification fires at the moment of account creation, when newcomers are trying to complete the task that brought them to the wiki. An unread alert at that moment may pull attention away from that first task without providing actionable guidance, and the notification's benefit to newcomers has not been empirically validated.

Experiment details

Overview

• Experiment Name: Welcome Notification Holdback Experiment
• Tracking Key: welcome-notification-holdback
• Risk Level: Tier 3: Low Risk (only standard, already-instrumented events and metrics; no new sensitive data or experiment-wide contextual attributes planned)

Traffic

• Assign Variation by Attribute: MW User ID
• Traffic Percent: 100% (maximum allowed traffic for MW User ID assignment)
• Variations
  • Control (control): New account holders receive the current Welcome notification (Echo welcome notification) upon account creation, as they do today.
  • Holdback (no-welcome-notification): The Welcome notification is suppressed for new account holders. No other notifications or onboarding surfaces are changed.
• Targeting
  • MediaWiki Database Code in:
    • Growth team pilot wikis: arwiki, enwiki, eswiki, frwiki

• Expected start date: TBD

Metrics

• Goal metrics
  • Constructive activation
• Secondary metrics
  • Logged-in reader retention
• Guardrail metrics
  • Second week editor retention

Needs to be updated / added to this task:

1. Instrumentation spec
2. Expected start date
3. GrowthBook links for all three metrics

Open questions:

• Is this technically feasible? (A/B testing with notifications has proven challenging in the past)
• Is this something we can experiment with as a separate experiment, or should we attempt to bundle this into the broader Account Set up experiment? T432126: Experiment: DE1.3.1 Early onboarding & account set-up