Page MenuHomePhabricator

New Discussion Tool: calculate workflow engagement metrics
Open, MediumPublic

Description

Purpose

This task is about analyzing how people are engaging with the New Discussion Tool to...

Help answer these questions:

  1. Are Junior Contributors able to use the New Discussion Tool successfully?
  2. Are Junior Contributors using the tool in ways that negatively impact the experience of others?

So we can determine:
What – if any – changes should be made to the New Discussion Tool before measuring the impact of it on peoples' likelihood to add new discussion topics to talk pages.

Workflow engagement metrics

1. Are Junior Contributors able to use the New Discussion Tool successfully?

  • Of the people who click the new discussion tool link (action = 'init'), what % of people reach the following steps of the edit workflow?
    • what % of people see the interface ready (action = 'ready')
    • what % of people start typing a comment (action = 'firstChange')
    • what % of people click the Reply button (action = 'saveIntent')
    • what % of people successfully publish the comment they were drafting (action = 'saveSuccess')?
  • How does the percent completion rate (% of people successfully publish the comment they were drafting (action = 'saveSuccess')) for edits made with the new discussion tool compare to the percent completion rate of edits made using the existing new section workflow by input mode – source / visual ?

2. Are Junior Contributors finding the tool valuable?

  • Percent of people who publish at least one comment with the new discussion tool and explicitly turn off the feature in Special:Preferences.

2. Are Junior Contributors using the tool in ways that negatively impact the experience of others?

  • The percentage of all edits made with the New Discussion Tool that are reverted within 48 hours of being published.
  • The number of users blocked after starting new sections on a talk page

Meta: For each of the above metrics, we will provide a breakdown by wiki where the New Discussion Tool is currently deployed. We will also provide a breakdown by the following experience levels to compare how the workflow engagement of Junior Contributors compares to Senior Contributors:

  • Under 100 cumulative edits (Junior Contributors)
  • Over 500 cumulative edits (Senior Contributors)

Done

  • We've agreed on the metrics we will use to answer the questions listed in the Workflow engagement metrics section above.
  • An analysis is completed that contains the metrics we'll have defined in the Workflow engagement metrics section above.

Note: Because the New Discussion Tool and the Reply Tool are built atop the same internals, there is no way to look at events in isolation and know the tool that was used to emit the event. This means, to look exclusively at the New Discussion Tool behavior, we'll need to join VEFeatureUse with EditAttemptStep.

Event Timeline

ppelberg renamed this task from New Discussion Tool: calculate workflow engagement metrics to New Discussion Tool: conduct impact analysis .Dec 2 2020, 1:54 AM
ppelberg renamed this task from New Discussion Tool: conduct impact analysis to New Discussion Tool: calculate workflow engagement metrics .
ppelberg updated the task description. (Show Details)
ppelberg updated the task description. (Show Details)

META
I've removed the specific metrics from the task description's Workflow engagement metrics questions per @MNeisler and I coming to agree that it's best for Megan to define the metrics and I the questions the metrics are intended to help us answer.

MNeisler triaged this task as Medium priority.Feb 11 2021, 9:18 PM
MNeisler moved this task from Next 2 weeks to Needs Sign-off on the Product-Analytics (Kanban) board.

@ppelberg - I updated the task description's Workflow engagement metrics section with suggested metrics.

Note I tried to keep this list focused on metrics that would help provide key insights to answer the identified questions. If the initial results are unclear or reveal an anomaly, we can always look at additional breakdowns of the data to help provide further insights into any identified trends.

Let me know if you have any questions or proposed changes.

Note I tried to keep this list focused on metrics that would help provide key insights to answer the identified questions. If the initial results are unclear or reveal an anomaly, we can always look at additional breakdowns of the data to help provide further insights into any identified trends.

Understood and this makes sense to me.

Let me know if you have any questions or proposed changes.

The metrics you have proposed in the task description look good.

A question: can we include a metric that helps us know the percentage of Junior and Senior Contributors who use the New Discussion Tool to add at least one topic and then turn off the feature?

Thinking: knowing if a relatively large number of people are turning off the tool after using it will serve as a helpful leading indicator for issues people may be having with the tool we are not aware of.

A question: can we include a metric that helps us know the percentage of Junior and Senior Contributors who use the New Discussion Tool to add at least one topic and then turn off the feature?
Thinking: knowing if a relatively large number of people are turning off the tool after using it will serve as a helpful leading indicator for issues people may be having with the tool we are not aware of.

Yes. I agree that would be valuable to include. I was originally thinking we should track that as part of the Adoption Metrics T263053 as a measure of disruption but I think it would useful to track in both analyses. It's a good indicator of if people are finding the tool disruptive to their experience (adoption metric) and/or if they find the tool valuable (engagement metric)

I went ahead and added this metric to the description. Let me know if you have any other questions or suggested changes.

A question: can we include a metric that helps us know the percentage of Junior and Senior Contributors who use the New Discussion Tool to add at least one topic and then turn off the feature?
Thinking: knowing if a relatively large number of people are turning off the tool after using it will serve as a helpful leading indicator for issues people may be having with the tool we are not aware of.

Yes. I agree that would be valuable to include. I was originally thinking we should track that as part of the Adoption Metrics T263053 as a measure of disruption but I think it would useful to track in both analyses. It's a good indicator of if people are finding the tool disruptive to their experience (adoption metric) and/or if they find the tool valuable (engagement metric)

Great and tracking in both places sounds good.

I went ahead and added this metric to the description. Let me know if you have any other questions or suggested changes.

Excellent. I think the scope (read: task description) is done. I'm assigning this task back over to you to work on sometime next quarter (Q4).