INSTRUMENTATION SPECIFICATION

User story & summary

As a Growth team member, I want instrumentation in place and validated via an A/A test so that we can reliably measure the DE1.3.1 experiment before the A/B test begins.

Background

Parent task: T430058: [EPIC] Account Setup (DE1.3.1 - early onboarding)
Experiment details: T432126: Experiment: DE1.3.1 Early onboarding & account set-up

Hypothesis: If we reduce cognitive load and provide more personalized post-account creation experiences for new account holders, then a greater proportion of users will constructively activate by making their first edit.

Details

Events/interactions to instrument:

• Welcome Survey / Account Setup started (at least one answer completed)
• Welcome Survey / Account Setup completed (all questions completed)
• Proportion of accounts with email addresses
• Homepage impression

Which schema(s)/stream(s) to use: TBD

This A/A test should validate:

• Instrumentation is working as expected.
• There are no SRM between groups. (Although this might not be verifiable until an A/B test is started)

Testable hypothesis

If we run an A/A test with identical experiences in both groups, it will show no significant differences between groups because any observed difference would indicate a problem with instrumentation or group assignment (SRM).

Experiment details

Overview

• Experiment Name: Account setup A/A test (DE1.3.1)
• Tracking Key: TBD
• Risk Level: Low risk

Traffic

• Assign Variation by Attribute: MW User ID
• Traffic Percent: 100%
• Variations:
  • A1 (control): default post-account-creation experience
  • A2 (treatment): identical to control (A/A test — no experience differences between groups)
• Targeting
  • MediaWiki Database Code in:
    • arwiki, enwiki, frwiki, eswiki
• Expected start date: August 24, 2026

Metrics

• Goal metrics:
  • Second week (editor) retention => T431016: Move DE1.3 (editor second-week retention) into GrowthBook
  • For REx: internal referrer rate => T434837: anyPageVisit: record referrer class in page_visit events' action_source
• Secondary metrics:
  • Constructive Activation => T433322: [DE 1.1.9] Define constructive edit-based metrics in GrowthBook
  • Welcome Survey / Account Setup started
  • Welcome Survey / Account Setup completed
  • Proportion of accounts with email addresses
• Guardrail metrics:
  • Homepage impression

• Instrumentation specs for DE1.3.1 are drafted and agreed upon with @KStoller-WMF and @MNeisler
• Instrumentation is set up to support T432126: Experiment: DE1.3.1 Early onboarding & account set-up