Purpose

This task is about analyzing how people are engaging with the New Discussion Tool to...

Help answer these questions:

1. Are Junior Contributors able to use the New Discussion Tool successfully?
2. Are Junior Contributors using the tool in ways that negatively impact the experience of others?

So we can determine:
What – if any – changes should be made to the New Discussion Tool before measuring the impact of it on peoples' likelihood to add new discussion topics to talk pages.

Workflow engagement metrics

1. Are Junior Contributors able to use the New Discussion Tool successfully?

• Of the people who click the new discussion tool link (action = 'init'), what % of people reach the following steps of the edit workflow?
  • what % of people see the interface ready (action = 'ready')
  • what % of people start typing a comment (action = 'firstChange')
  • what % of people click the Reply button (action = 'saveIntent')
  • what % of people successfully publish the comment they were drafting (action = 'saveSuccess')?
• How does the percent completion rate (% of people successfully publish the comment they were drafting (action = 'saveSuccess')) for edits made with the new discussion tool compare to the percent completion rate of edits made using the existing new section workflow by input mode – source / visual ?

2. Are Junior Contributors finding the tool valuable?

• Percent of people who publish at least one comment with the new discussion tool and explicitly turn off the feature in Special:Preferences.

2. Are Junior Contributors using the tool in ways that negatively impact the experience of others?

• The percentage of all edits made with the New Discussion Tool that are reverted within 48 hours of being published.
• The number of users blocked after starting new sections on a talk page

Meta: For each of the above metrics, we will provide a breakdown by wiki where the New Discussion Tool is currently deployed. We will also provide a breakdown by the following experience levels to compare how the workflow engagement of Junior Contributors compares to Senior Contributors:

• Under 100 cumulative edits (Junior Contributors)
• Over 500 cumulative edits (Senior Contributors)

Done

• We've agreed on the metrics we will use to answer the questions listed in the Workflow engagement metrics section above.
• An analysis is completed that contains the metrics we'll have defined in the Workflow engagement metrics section above.

Note: Because the New Discussion Tool and the Reply Tool are built atop the same internals, there is no way to look at events in isolation and know the tool that was used to emit the event. This means, to look exclusively at the New Discussion Tool behavior, we'll need to join VEFeatureUse with EditAttemptStep.