Page MenuHomePhabricator

[SPIKE] Investigate how frequently newcomers make an edit attempt at en.wiki
Closed, ResolvedPublic

Description

Decision(s) to be made

  • Is it feasible for us to consider evaluating the impact of Edit Check through an A/B test at English Wikipedia where visual editor is not yet available to newcomers by default

Requirements

  • Provide data on the number and proportion of newcomers at English Wikipedia who complete a new content edit using Visual Editor. Review weekly rates.
  • Include breakdown by platform (mobile web and desktop) and type of user: unregistered, newcomers (first edit), and users with 100 or fewer edits.
  • Compare observed weekly rates to rates on other large Wikipedias where VE is offered as default on mobile.

Story

@ppelberg to populate.

Open question

  • 1. How long might an A/B test at English Wikipedia that centers newcomers and Junior Contributors need to run to reach statistically significant results

Event Timeline

MNeisler triaged this task as Medium priority.
MNeisler edited projects, added Product-Analytics (Kanban); removed Product-Analytics.
MNeisler moved this task from Next 2 weeks to Doing on the Product-Analytics (Kanban) board.

@ppelberg See summary below of the rates at which newcomers publish Visual Editor edits on English Wikipedia. The data reflects edits published on English Wikipedia on a content namespace over a 2-week time frame in May 2025:

Some initial takeaways

Based on these rates, we would have data on mobile new content VE edits by about 150 newcomers on English Wikipedia after 4 weeks. This should be sufficient to evaluate any large effects of Edit Check on these types of edits through an AB test (similar to the size effects observed in the previsous Reference Check AB Test). A longer duration would be needed to detect any smaller changes between the two groups.

We would have sufficient data after 4 weeks to evaluate both small and large overall effects (across both platforms and all eligible experience levels ); however, since VE is not available as default to newcomers on mobile at enwiki, I'd recommend that we evaluate this group separately in the AB Test as the newcomer workflow for completing a VE new content edit on English Wikipedia will be different.

Results Summary

Proportion of newcomers at English Wikipedia who complete a new content edit using Visual Editor.
Overall, about 2.5% of all newcomers (431 distinct editors) who published an edit completed a new content edit using VE on English Wikipedia. On mobile web, only 1.3% of newcomers (69 users) completed a new content edit using VE over a two-week timeframe.

By platform and user experience:

Desktop VE new content edits by user edit count

experience_level_groupNumber of editorsProportion of editors
Newcomer3623.1%
Junior Contributor12904.2%

Mobile VE new content edits by user edit count

experience_level_groupNumber of editorsProportion of editors
Newcomer691.3%
Junior Contributor2914.2%

Note: We don't have data on the number distinct unregistered editors but there were 513 mobile and 953 desktop new content edits completed with VE by unregistered users.

Comparison to newcomer mobile VE new content rates on other larger Wikipedias
While a higher proportion of newcomers publish edits with VE on mobile on French Wikipedia and Spanish Wikipedia, the number of distinct newcomers is similar due to the larger size of English Wikipedia.

wikiType of VE editNumber of editorsProportion of edits
enwikiVE non-content121523.4%
enwikiVE content691.3%
eswikiVE non-content51662.5%
eswikiVE content485.8%
frwikiVE non-content30445.8%
frwikiVE content629.3%

A couple notes:

  • A new content edit defined as all edits with the editcheck-newcontent tag. See Edit check/Tags definition
  • Newcomer is defined as a user making their first edit on that Wikipedia.
  • I looked at published edits vs. edit attempts as the majority of the metrics we use to evaluate Edit Check are based on the published edit status (For example, was the edit published with a referece, was it reverted?). We would need a sufficient sample of published edits to evaluate these metrics.

@MNeisler the findings you share in T394951#10906272 equip us with the info we need to specify how long a controlled experiment at en.wiki will need to run to return statistically significant results.

The work of finalizing the proposing we'll share with volunteers will happen in T400103.