Page MenuHomePhabricator

Add more granular experience level buckets to New Topic Tool A/B test report
Closed, ResolvedPublic

Description

This task represents the work involved with adding more granular experience level buckets to the New Topic Tool A/B test report published in T277825#7846522 s that we can, as Megan put it well, "...isolate this observed decrease in New Tool completion rate to a more specific level of experience."

Decision to be made

The additional granularity this task is asking for will help the Editing Team decide the people for whom we will propose the New Topic Tool be made available by default as part of T290517.

Requirements

Done

  • A link to the updated report that meets the ===Requirements above is posted to this ticket

Event Timeline

ppelberg added a project: DiscussionTools.
ppelberg moved this task from Backlog to Triaged on the DiscussionTools board.
ppelberg moved this task from Backlog to Analytics on the Editing-team (Tracking) board.
MNeisler edited projects, added Product-Analytics (Kanban); removed Product-Analytics.
MNeisler moved this task from Doing to Needs Sign-off on the Product-Analytics (Kanban) board.

To further investigate the impact of experience level on Senior Contributors' new topic tool usage, I broke down experience level into the following edit segments (buckets) [i]: '0-100 edits', (Junior Contributors), '101-500 edits', '501-1000 edits', '1001-1500 edits', '1501-2000 edits', '2001-2500 edits', '2501-3000 edits', 'over 3000 edits'.

See summary below and updated report for details.

New Topic Completion Rate

new_topic_completes_byexp (1).png (2×4 px, 330 KB)

By breaking down the experience levels into more granular segments, we are able to isolate the decrease in completion rate to contributors with over 2500 edits [ii].

  • There is a higher new topic tool completion rate for all experience groups under 2500 cumulative edits, with percent increases ranging from only 0.4% for the 1501-2000 edit group level to a 54.5% increase for the 1001-1500 edit count group.
  • There is a see a lower new topic tool completion rate for users with over 2500 cumulative edits (3.4% decrease for the 2501-3000 edit count group and 10.7% decrese for the over 3000 edit count group).

Revert rates and block rates:

  • Unfortunately, it is difficult to accurately compare revert rates or block rates with much confidence at these more granular experience levels as there are very few of these events logged for users at higher experience levels, especially for the existing add new section link editing method which was sampled. The low number of events logged creates more uncertainty in the observed rates being representative of the actual population.
  • Excluding the edit groups where there is not sufficient evidence to compare, we are able to identify a slight increase in reverts for the over 3000 edits group (1.36% to 1.86%).

[i]Rationale for Selected Edit Segment Widths: I tried to make the segment widths granular enough to provide a higher level of detail on the majority of users while also capturing some of the high powers users; however, I needed to do this without over restricting the sample. If we make the segments too granular, there are not be enough events logged within that group to have a representative sample. This is especially true for higher edit count groups where there are fewer users.
[ii] Red Error Bars: The red error bars in the charts indicate the level of uncertainty in the identified completion rate. There is more uncertainty in the identified completion rates for higher edit count contributors using the existing add new section link. This is due to the smaller sample size available for in these groups (There are fewer users in the higher level edit counts and these events are sampled at 6.25%).

@ppelberg - Reassigning to you to review. Let me know if you have any questions.

See summary below and updated report for details.
...@ppelberg - Reassigning to you to review. Let me know if you have any questions.

The analysis you shared above looks great, @MNeisler. No further questions from me.

I've shared these findings on mediawiki.org in this edit: https://w.wiki/58aM.