Page MenuHomePhabricator

Workflow engagement check: Replying v1.0
Closed, DuplicatePublic

Description

As mentioned in T246679, this task is about analyzing the "Workflow engagement metrics" below in order to help answer the following questions:

  1. Are people are having success using the feature?
  2. Are people using the tool in ways that negatively impact the experience of others?

Answering the questions above will help us determine whether we need to make changes to the workflow beyond those already planned for V2.0: T235593.

Work on this task can begin 2-3 weeks after v1.0 is deployed as an opt-out user Preference (read: after T249394 is resolved).

Open questions

  • How long after T245794 is resolved should this analysis happen?
    • We are planning for this work to begin 2-3 weeks after T249394 is resolved.
  • Is additional instrumentation needed to calculate the % of comments posted using the Replying workflow that are reverted?
  • Is additional instrumentation needed to calculate the # of people blocked after posting a reply using DiscussionTools?
  • Is additional instrumentation needed to calculate the % of comments posted using Replying workflow that are made in "good faith"?
    • Potentially. If it additional instrumentation is required we are not going to prioritize adding it at this time. See: T247138#6006735.

Workflow engagement metrics

This section outlines the information that will help the team decide whether the tool should be deployed to more people and whether changes should be made to the workflow beyond those already planned for V2.0.

1. Are people publishing the replies they start?

  • % of people who click reply AND who start typing a comment
  • % of people who start typing a comment AND who publish their comment to page

2. Are people abandoning the replies they start?

  • % of people who start typing a comment AND who actively abandon their comment

3. Are people satisfied with the feature?

  • % of people who publish at least one comment and turn off DiscussionTools in Beta Features
  • Of the people who have made at least 1 edit with the Replying workflow, the % of people who posted 1, 2-5, 5-10 and 10+ comments

4. Are people posting disruptive comments?

  • % of comments posted using Replying workflow that are reverted
  • # of people blocked after posting a reply using DiscussionTools
  • % of comments posted using Replying workflow that are made in good faith

Done

  • Analysis is completed that contains the "Workflow engagement metrics" listed above

Event Timeline

ppelberg created this task.Mar 6 2020, 10:31 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 6 2020, 10:31 PM
ppelberg updated the task description. (Show Details)Mar 7 2020, 12:51 AM
ppelberg added a subscriber: Mayakp.wiki.
ppelberg updated the task description. (Show Details)Mar 7 2020, 12:56 AM
ppelberg updated the task description. (Show Details)
ppelberg updated the task description. (Show Details)Mar 7 2020, 1:05 AM

@Mayakp.wiki and I need to discuss the scope of this ticket before it ready to be worked on.

SNowick_WMF triaged this task as Medium priority.Mar 10 2020, 5:33 PM
SNowick_WMF moved this task from Triage to Backlog on the Product-Analytics board.
SNowick_WMF moved this task from Backlog to Current Quarter on the Product-Analytics board.

@Mayakp.wiki and I need to discuss the scope of this ticket before it ready to be worked on.

Meeting notes: 11-March
Some metrics included in the task description, may require instrumentation that we do not currently have in place. Those metrics are:

6. Are people posting disruptive comments?

  • % of comments posted using Replying workflow that are reverted
  • # of people blocked after posting a reply using DiscussionTools
  • % of comments posted using Replying workflow that are made in good faith

Next steps
@Mayakp.wiki to determine whether the metrics listed below require additional instrumentation in order to compute:

  • % of comments posted using Replying workflow that are reverted
  • # of people blocked after posting a reply using DiscussionTools
  • % of comments posted using Replying workflow that are made in good faith
ppelberg reassigned this task from ppelberg to Mayakp.wiki.Mar 11 2020, 7:43 PM

Update: 18-March
We can assume this analysis will not happen this quarter (read: before 30-March).

  • Rationale: this analysis depends on T245794 being deployed and T247152 being completed, both of which are unlikely to happen in the next two weeks.

Next steps
@Mayakp.wiki to determine whether the metrics listed below require additional instrumentation in order to compute:

  • % of comments posted using Replying workflow that are reverted
  • # of people blocked after posting a reply using DiscussionTools
  • % of comments posted using Replying workflow that are made in good faith

Update: 27-March
After meeting with @Mayakp.wiki this week, it sounds like we may not be able to rely on ORES edit quality models to determine the quality of edits made on talk pages.

If this turns out to be true, we will use the following measures, ordered by importance:

  • % of comments posted using Replying workflow that are reverted
  • # of people blocked after posting a reply using DiscussionTools
Mayakp.wiki added a comment.EditedMar 30 2020, 5:40 PM

Next steps
@Mayakp.wiki to determine whether the metrics listed below require additional instrumentation in order to compute:

  • % of comments posted using Replying workflow that are reverted

In order to calculate this, we will need to use the Revision ID field in editattemptschema only for events having action=saveSuccess.

With this in mind, T244939 highlights that the null revision ID is mostly in events having action = ready, loaded, abort, firstchange etc. Hence, fixing T244939 is not high priority for getting % of comments posted using Replying workflow that are reverted

ppelberg updated the task description. (Show Details)Mar 30 2020, 7:25 PM

Next steps
@Mayakp.wiki to determine whether the metrics listed below require additional instrumentation in order to compute:

  • % of comments posted using Replying workflow that are reverted

In order to calculate this, we will need to use the Revision ID field in editattemptschema only for events having action=saveSuccess.

With this in mind, T244939 highlights that the null revision ID is mostly in events having action = ready, loaded, abort, firstchange etc. Hence, fixing T244939 is not high priority for getting % of comments posted using Replying workflow that are reverted.

This is good to hear; thank you for getting to the bottom of this, @Mayakp.wiki.

I've reflected the above in the "Open questions" portion of the task description.

ppelberg added a comment.EditedApr 1 2020, 7:20 PM

Additional instrumentation needed for blocked users

Next steps
@Mayakp.wiki to determine whether the metrics listed below require additional instrumentation in order to compute:

  • # of people blocked after posting a reply using DiscussionTools

Today, @Mayakp.wiki confirmed additional instrumentation is NOT required to measure the # of people blocked after posting a reply using DiscussionTools.

This is now reflected in the task description.

ppelberg updated the task description. (Show Details)Apr 1 2020, 7:20 PM
ppelberg updated the task description. (Show Details)Apr 1 2020, 8:14 PM

Revisions to engagement metrics
During our meeting on Wednesday, 1-April, @Mayakp.wiki and I confirmed the following changes to the metrics we will calculate to help evaluate how people are using the new Replying tool:

  • REMOVE: Distribution of elapsed times between when a person starts typing their comment and when they publish their comment
    • Rationale: we think this metric is likely to be noisy.
  • REMOVE: "Are people needing to correct mistakes the tool produced?"
    • Reasons: we think this metric is likely to be noisy as well (e.g. what if someone is just correcting a typo?). Also: at this stage, we can more reliably determine this by manually reviewing edits made using the DiscussionTools via change tags.

The task description has been updated to reflect these changes.

ppelberg updated the task description. (Show Details)Apr 4 2020, 12:54 AM
ppelberg updated the task description. (Show Details)Apr 4 2020, 1:30 AM
ppelberg updated the task description. (Show Details)Apr 8 2020, 7:10 PM