Page MenuHomePhabricator

New Discussion Tool: calculate workflow engagement metrics
Open, Needs TriagePublic

Description

Purpose

This task is about analyzing how people are engaging with the New Discussion Tool to...

Help answer these questions:

  1. Are Junior Contributors able to use the New Discussion Tool successfully?
  2. Are Junior Contributors using the tool in ways that negatively impact the experience of others?

So we can determine:
What – if any – changes should be made to the New Discussion Tool before measuring the impact of it on peoples' likelihood to add new discussion topics to talk pages.

Workflow engagement metrics

1. Are Junior Contributors able to use the New Discussion Tool successfully?

  • New Discussion Tool funnel (grouped by experience level and the input mode – source / visual – they are shown by default):
    • Of the people who open the New Discussion Tool (action = 'init_type:section; integration = ??? ), what % of people see the interface ready (action = 'ready')?
    • Of the people who see the interface is ready action = 'ready, what % of people start drafting a new topic (action = 'firstChange')?
    • Of the people who start typing a new topic (action = 'firstChange'), what % of people click the Add topic button (action = 'saveIntent')?
    • Of the people who click the Add topic button (action = 'saveIntent'), what % of people successfully publish the topic they were drafting (action = 'saveSuccess')?
  • New Discussion Tool topic completion (grouped by experience level and the input mode – source / visual – they are shown by default):
    • Of the people who open the New Discussion Tool (action = 'init_type:section), what % of people successfully publish the topic they were drafting (action = 'saveSuccess')?
  • As-is section=new "Add topic" worfklow completion rate (grouped by experience level):
    • Of the people who open the as-is new section form (action = 'init_type:section;integration = ??? ), what % of people successfully save an edit (action = 'saveSuccess')?

2. Are Junior Contributors using the tool in ways that negatively impact the experience of others?

  • The number and percentage of people who have used the New Discussion tool to add at least one new topic and then explicitly turn off the feature, grouped by experience level and wiki
  • The number and percentage of all edits made with the New Discussion Tool that are reverted within 48 hours of being published.
  • The number of users blocked after using the New Discussion Tool to add a topic.

Open questions/Todo

  • Verify all event names in the === Workflow engagement metrics section above.

Note: Because the New Discussion Tool and the Reply Tool are built atop the same internals, there is no way to look at events in isolation and know the tool that was used to emit the event. This means, to look exclusively at the New Discussion Tool behavior, we'll need to join VEFeatureUse with EditAttemptStep.

Event Timeline

ppelberg created this task.Sep 16 2020, 7:39 PM
ppelberg renamed this task from New Discussion Tool: calculate workflow engagement metrics to New Discussion Tool: conduct impact analysis .Dec 2 2020, 1:54 AM
ppelberg renamed this task from New Discussion Tool: conduct impact analysis to New Discussion Tool: calculate workflow engagement metrics .
ppelberg updated the task description. (Show Details)