@Mayakp.wiki and I need to discuss the scope of this ticket before it ready to be worked on.

In T247138#5950519, @ppelberg wrote:

@Mayakp.wiki and I need to discuss the scope of this ticket before it ready to be worked on.

Meeting notes: 11-March
Some metrics included in the task description, may require instrumentation that we do not currently have in place. Those metrics are:

6. Are people posting disruptive comments?

• % of comments posted using Replying workflow that are reverted
• # of people blocked after posting a reply using DiscussionTools
• % of comments posted using Replying workflow that are made in good faith

Next steps
@Mayakp.wiki to determine whether the metrics listed below require additional instrumentation in order to compute:

• % of comments posted using Replying workflow that are reverted
• # of people blocked after posting a reply using DiscussionTools
• % of comments posted using Replying workflow that are made in good faith

Update: 18-March
We can assume this analysis will not happen this quarter (read: before 30-March).

• Rationale: this analysis depends on T245794 being deployed and T247152 being completed, both of which are unlikely to happen in the next two weeks.

Next steps
@Mayakp.wiki to determine whether the metrics listed below require additional instrumentation in order to compute:

• % of comments posted using Replying workflow that are reverted
• # of people blocked after posting a reply using DiscussionTools
• % of comments posted using Replying workflow that are made in good faith

Update: 27-March
After meeting with @Mayakp.wiki this week, it sounds like we may not be able to rely on ORES edit quality models to determine the quality of edits made on talk pages.

If this turns out to be true, we will use the following measures, ordered by importance:

• % of comments posted using Replying workflow that are reverted
• # of people blocked after posting a reply using DiscussionTools

Next steps
@Mayakp.wiki to determine whether the metrics listed below require additional instrumentation in order to compute:

• % of comments posted using Replying workflow that are reverted

In order to calculate this, we will need to use the Revision ID field in editattemptschema only for events having action=saveSuccess.

With this in mind, T244939 highlights that the null revision ID is mostly in events having action = ready, loaded, abort, firstchange etc. Hence, fixing T244939 is not high priority for getting % of comments posted using Replying workflow that are reverted

In T247138#6011729, @Mayakp.wiki wrote:

Next steps
@Mayakp.wiki to determine whether the metrics listed below require additional instrumentation in order to compute:

• % of comments posted using Replying workflow that are reverted

In order to calculate this, we will need to use the Revision ID field in editattemptschema only for events having action=saveSuccess.

With this in mind, T244939 highlights that the null revision ID is mostly in events having action = ready, loaded, abort, firstchange etc. Hence, fixing T244939 is not high priority for getting % of comments posted using Replying workflow that are reverted.

This is good to hear; thank you for getting to the bottom of this, @Mayakp.wiki.

I've reflected the above in the "Open questions" portion of the task description.

Additional instrumentation needed for blocked users

In T247138#6006735, @ppelberg wrote:

Next steps
@Mayakp.wiki to determine whether the metrics listed below require additional instrumentation in order to compute:

• # of people blocked after posting a reply using DiscussionTools

Today, @Mayakp.wiki confirmed additional instrumentation is NOT required to measure the # of people blocked after posting a reply using DiscussionTools.

This is now reflected in the task description.

Revisions to engagement metrics
During our meeting on Wednesday, 1-April, @Mayakp.wiki and I confirmed the following changes to the metrics we will calculate to help evaluate how people are using the new Replying tool:

• REMOVE: Distribution of elapsed times between when a person starts typing their comment and when they publish their comment
  • Rationale: we think this metric is likely to be noisy.
• REMOVE: "Are people needing to correct mistakes the tool produced?"
  • Reasons: we think this metric is likely to be noisy as well (e.g. what if someone is just correcting a typo?). Also: at this stage, we can more reliably determine this by manually reviewing edits made using the DiscussionTools via change tags.

The task description has been updated to reflect these changes.