Page MenuHomePhabricator

Who should be invited to view the moderator homepage experiment?
Closed, ResolvedPublic

Description

Our moderator homepage experiment is primarily geared towards newer contributors - editors who have made some edits, and understand the fundamentals of contributing, but have not been editing for a long time; they might not have reverted an edit, or participated in a noticeboard discussion, for example.

There are two ways we could target users to view the new homepage - a popup in the Newcomer homepage, and and invite outside of the Newcomer homepage. These don't necessarily need to sample users in the same way, and I think we might want to do both simultaneously.

We need to set criteria that are stringent enough that we don't capture users who are either too early or too late in their editing journey, while being broad enough that we can collect data and feedback from a sufficient number of users.

See T402647: Launch new moderator homepage MVP for details on the wikis we are considering targeting. We may only focus on the top few wikis by size from this list, for localization reasons (many of the wikis here are also considerably smaller than the biggest few, so may not be worth our while).

This ticket is for capturing our work towards figuring out what the boundaries should be on this invite.

Event Timeline

@MNeisler I was going to create a subtask but I think we can just use this one. Per our conversation, let's start by getting the number of users who made at least one edit in the last 4 weeks (I think we'll aim to run our experiment for about this long) and:

  • Have made 100 - 1000 edits
  • Registered their account no more than 2 years ago.

And let's start with the following wikis so we can ballpark what these figures look like at different size wikis: id.wiki, tr.wiki, az.wiki, te.wiki, is.wiki

We might then iterate the numbers above to see if other edit counts/tenures produce meaningfully different results.

And let's start with the following wikis so we can ballpark what these figures look like at different size wikis: id.wiki, tr.wiki, az.wiki, te.wiki, is.wiki

Per T402917 let's actually run this analysis on id.wiki, tr.wiki, simple.wiki, bn.wiki, az.wiki.

@Samwalton9-WMF
I reviewed the number of distinct editors that met the following criteria at id.wiki, tr.wiki, simple.wiki, bn.wiki, az.wiki.:

  • Made at least one edit in the last 4 weeks. Note: In this analysis, this included any edits completed from 1 August through 29 August 2025.
  • Have made 100 - 1000 edits
  • Registered their account no more than 2 years ago.

I also reviewed some slight iterations of the criteria (e.g. extending user edit count ranges or registration time frame) to see if other edit counts/tenures produce meaningfully different results.

Results
Results on number of users that meet each criteria are currently documented in this google sheet for review. Some of the data is too granular and will need to be scrubbed prior to sharing publicly. For each wiki, data is provided on the total number of users that meet the criteria and by platform (mobile web editor vs desktop editors).

Initial Insights
Population sizes were small when limited to the initial proposed criteria. Overall, changing registration timeframes had minimal impact on the number of users, but increasing the cumulative edit count range to include lower edit counts (especially for smaller wikis) caused a larger increase on the population of users. Let me know if there are any other iterations you'd like to explore.

Of the variations reviewed so far, Criteria 5 (Increasing edit count range to 10 -1000 edits) achieved the highest number of users of all the criteria explored.

Methodology Notes:
Retrieved editing data from mediawiki_history and user registration data from mediawiki_user_history (augmented with ServerSideAccountCreation data). Bots were excluded and I also limited to edits completed on a main namespace.

We discussed the relatively low number of users that meet our criteria on the wikis we've been looking at. Let's re-run the analysis including some more wikis where we can deploy the Revert Risk model via the ORES extension, and open our criteria up further:

  • vi.wiki
  • th.wiki
  • no.wiki
  • el.wiki
  • hi.wiki
  • 100+ edits
  • <3 years tenure
  • 8 weeks
  • Anyone who made an edit

@Samwalton9-WMF

I've re-run the analysis with the following criteria:

  • Made at least one edit in the last 8 weeks.
  • Have made 100+ cumulative edits
  • Registered their account no more than 3 years ago.
  • Wikis: viwiki, thwiki, nowiki, elwiki, hiwiki, idwiki, trwiki, simplewiki, bnwiki, azwiki

Results:
The extended criteria increase the total number of distinct editors to ~700 users across all ten projects. Per wiki data is documented in this google sheet ("updated analysis 31 Oct" tab). Similar to before, data is provided on the total number of users that meet the criteria and by platform (mobile web editor vs desktop editors) at each Wikipedia.

Let me know if you have any questions or need any adjustments to the above criteria.

Thanks, this is great!