Page MenuHomePhabricator

Publish experiment results for "Account Creation: No Benefits on desktop" (WE1.8 FY25/26)
Closed, ResolvedPublic

Description

User Story:

As a Product Manager, I want to remove the benefits block from the desktop account creation form, so that we reduce UI complexity for new users and can delete the non-trivial code that maintains it, consistent with results from our mobile form simplification experiments.

As a Wikimedia community member, I want to be informed about changes to the account creation experience and assured that they are backed by evidence, so that I understand why a change was made.

Background:

Prior mobile Account Creation Experiments (1.8.3) showed that simplifying the account creation form, including hiding the benefits block, improved registration completion. This experiment tested whether removing the benefits block on desktop has any negative effect. Treatment was identical to control except the benefits block was removed. If removal is safe on all platforms, we can delete a substantial amount of non-trivial code.

Acceptance Criteria:

Event Timeline

Experiment results published here: https://www.mediawiki.org/wiki/Contributors/Account_Creation_Experiments#V4_Test_-_%22Benefits_Block%22_on_Desktop_(T430785)


"Benefits Block" on Desktop Experiment:

Hypothesis

Removing the benefits block from the desktop Special:CreateAccount form will not reduce the permanent account creation rate. Based on prior mobile A/B test results, we expect the rate to hold steady or improve slightly.

What we tested
Prior mobile account creation experiments (1.8.3) showed that simplifying the account creation form, including hiding the benefits block, improved registration completion. This experiment tested whether removing the benefits block on desktop has any negative effect. Treatment was identical to control except the benefits block was removed. If removal is safe on all platforms, we can delete a substantial amount of non-trivial code.

  • A/B Test Duration: Jul 9 – Jul 27, 2026 (19 days)
  • Traffic: English Wikipedia, 50% of eligible traffic
  • Control: 9,734 subjects
  • Treatment: 9,848 subjects

What we found
The guardrail held. Treatment showed a small, non-significant positive effect on permanent account creation rate (+0.33% relative, 55.8% chance of winning). We found no evidence of a regression or negative impact from removing the benefits block.

The secondary metric points in the same direction: create-account-button clickthrough showed a non-significant positive effect (+10.2% relative, 70.5% chance of winning).

Neither result is conclusive on its own, but both metrics moving in the same positive direction suggests that removing the benefits block may modestly improve account creation completion, and makes a meaningful regression unlikely. At this effect size and traffic level, confirming the guardrail result would require roughly a year of additional data.

Recommentation:
This experiment was designed around an expectation of no change or a very small improvement, and effects that small are inherently difficult to prove: the smaller the true difference between treatment and control, the larger the sample required to distinguish it from noise. Reaching significance here would take roughly a year of additional data, which is not a reasonable cost for a minor feature change. Given no detectable regression, a possible small benefit, and meaningful reductions in UI and code complexity, we can move forward with the treatment experience.

By the time someone lands on Special:CreateAccount, they have already decided to create an account. Our goal at that point is to remove distractions and streamline completion, not to persuade. The benefits block works against this in several ways:

Redundant content. The statistics display repeat figures already shown on the Main Page, so they add visual noise without new information.
Wrong place for the message. We should do more to connect new account holders to their impact and to the broader community, but the account creation form is not the right surface for that work. It belongs in post-registration touchpoints like onboarding and the Newcomer homepage.
Visual misalignment. The block's layout is inconsistent with the rest of the form, and the statistics are rendered in a font size larger than the primary "Create your account" call to action, drawing attention away from the one action we want users to complete.
Removing the block simplifies the page around its single purpose and lets us retire the non-trivial code that maintains it.

What happens next