In the V1 experiment on Special:CreateAccount on mobile, we, among other things, removed the benefits block for the treatment group (T415659) and saw a substantial improvement in the account creation rate.

Based on the same reasoning, and because dropping it everywhere would allow us to remove notable complexity from code and configuration, we would like to learn whether also dropping it on desktop at least does not have a strong negative on the account creation rate.

Measurement Plan

Experiment details

Experiment key and variation keys are in instrumentation specification: https://docs.google.com/spreadsheets/d/1Bo1gPny1NJkEtZYPaf3B3td4cjvtanJwX8FHHCyzcLc/edit?gid=0#gid=0

• Overview
  • Risk Level: Tier 3: Low Risk
• Traffic
  • Assign Variation by Attribute: Edge Unique (Cache non-splitting)
  • Traffic Percent: 100% of allowed traffic
• Targeting
  • MediaWiki Database Code in:
    • enwiki

Eligibility criteria

• user is anonymous
• user is on desktop (= not minerva skin)

Metrics

• Goal metrics
  • none. This was a non-inferiority test: the question is whether removal causes harm, not whether it drives improvement.
• Secondary metrics
  • Create Account Button CTR
• Guardrail metrics
  • permanent account creation rate

Stream: product_metrics.web_base

Contextual attributes

• performer_is_temp

Acceptance criteria

• create a follow-up task for experiment platform team to increase the enrollment percentage to 50% for that experiment
  • => T431132: Please allocate 50% traffic to "Account Creation: No Benefits on desktop" NCS experiment on enwiki
• run a simple NCS-experiment on enwiki's Special:CreateAccount on desktop where the only change between treatment and control is the benefits block being absent for treatment and present for control.