Page MenuHomePhabricator

Create response rate measurement plan
Closed, ResolvedPublic

Description

This task is about creating the measurement plan we will use to evaluate the impact of the notification interventions. See: T273920.

Done

  • Pre-mortem is complete (read: scenarios are documented and prioritized)

We've completed the pre-mortem and defined during which analysis (Adoption / Engagement / Impact) we will check on the quantitative leading indicators we've defined for the "MEDIUM" and "HIGH" priority scenarios. More in the Pre-mortem section of the measurement plan.

We've finalized the hypotheses and identified the KPI we will use for this set of interventions: Response rate as defined as the average time that elapses between when someone posts on a talk page and when a Junior Contributor posts a new comment in response. More in the Hypotheses section of the measurement plan.

We've defined three analyses that will comprise the topic subscription experiment. These analyses can be found in the Experiment plan of the measurement plan.

Tasks representing each of these analyses will be created as sub-tasks of T280895.

Event Timeline

Update: 10-Feb

The measurement plan is being drafted in this document (read only access for now): Talk pages project/Notifications/Measurement plan.

Immediate next steps

Guardrails
In this conversation on mediawiki.org, @Sdkb raised two potential consequences that could emerge when people are able to be more specific about the activity they are and are not made aware of:

  1. People could see a conversation, deem it to be not relevant enough for them to subscribe to in the moment, , and subsequently, miss out on a conversation that would've been valuable for them to be aware of.
  2. To evade "opponents," people editing in bad faith could start a new topic on a talk page to avoid the attention of people subscribed to a conversation ongoing elsewhere on the page.

I'm documenting these scenarios here so we can think about how we might monitor their prevalence.

24-Feb update
Per yesterday's conversation, @MNeisler is going to:

  • 1a) Review the "Hypotheses" and provide feedback if she thinks edits should be made to them
  • 1b) Propose quantitative metrics for each hypothesis, if applicable
  • 2a) Review the "Guardrails" and provide feedback if she thinks edits should be made to them
  • 2b) Propose quantitative metrics for each guardrail
MNeisler triaged this task as Medium priority.Mar 2 2021, 5:45 PM

I've reviewed and added my proposed quantitative metrics for the Hypotheses and Guardrails to the Talk pages project/Notifications/Measurement plan.

Reassigning to @ppelberg to clarify next steps.

Next steps:

  • 1. Team to assign impact value for each "Pre-mortem" scenario
  • 2. @ppelberg to prioritize scenarios
  • 3. @MNeisler to draft quantitative leading indiactors for applicable "pre-mortem" scenarios
  • 4. @MNeisler + @ppelberg' to finalize "Hypotheses" and KPIs," pending team review
  • 5. @MNeisler + @ppelberg' to draft/finalized "Experiment" plan

Next steps:

  • 1. Team to assign impact value for each "Pre-mortem" scenario
  • 2. @ppelberg to prioritize scenarios

The drafted pre-mortem can be found here (limited access for now): Talk pages project/Notifications/Pre-mortem.

  • 3. @MNeisler to draft quantitative leading indiactors for applicable "pre-mortem" scenarios

@MNeisler, for each of the "Scenarios" in this sheet [i] whose "Priority" has been marked as "HIGH" or "MEDIUM" can you please do the below?

  • 1. Add quantitative "Leading indicator" to applicable scenarios?
  • 2. Review the existing [Quant] "Leading indicator" I've suggested and comment any revisions you think should be made to them?

i. The contents of this spreadsheet will automatically be added to the Google Doc where we originally started documenting and evaluating the various pre-mortem scenarios.

@MNeisler, for each of the "Scenarios" in this sheet [i] whose "Priority" has been marked as "HIGH" or "MEDIUM" can you please do the below?

  1. Add quantitative "Leading indicator" to applicable scenarios?
    1. Review the existing [Quant] "Leading indicator" I've suggested and comment any revisions you think should be made to them?

@ppelberg - I've completed reviewing existing and adding quantifiable Leading indicators for all the high and medium tasks in the document as comments/suggestions. Let me know if you have any questions about the changes.

@ppelberg - I've completed reviewing existing and adding quantifiable Leading indicators for all the high and medium tasks in the document as comments/suggestions. Let me know if you have any questions about the changes.

Excellent.


Per the conversation @MNeisler and I had earlier today, this task can now be "Resolved" considering we have done the following (copied from the task's ===Done section):

  • Pre-mortem is complete (read: scenarios are documented and prioritized)

We've completed the pre-mortem and defined during which analysis (Adoption / Engagement / Impact) we will check on the quantitative leading indicators we've defined for the "MEDIUM" and "HIGH" priority scenarios. More in the Pre-mortem section of the measurement plan.

We've finalized the hypotheses and identified the KPI we will use for this set of interventions: Response rate as defined as the average time that elapses between when someone posts on a talk page and when a Junior Contributor posts a new comment in response. More in the Hypotheses section of the measurement plan.

We've defined three analyses that will comprise the topic subscription experiment. These analyses can be found in the Experiment plan of the measurement plan.

Tasks representing each of these analyses will be created as sub-tasks of T280895.

ppelberg updated the task description. (Show Details)