@MNeisler, copying over a few thoughts from conversations we've had before...

On the purpose of the metric(s) we decide on measuring for this fiscal year:

• We should have a metric that directly measures the impact of the interventions we deploy this year
• We should have a metric that measures a second order effect of our work

Measurement ideas (directly measurable)

• Increase the number of contributions by junior contributors to Wikipedia talk pages (article and user)
• Increase the number of junior contributors making contributions to Wikipedia talk pages (article and user)
• Increase the talk page "contribution rate" of junior contributors.
  • "Contribution rate" = % of junior contributors that start to post a reply or new discussion and follow through to post their "thought"
• Increase in junior contributors' level of engagement with talk pages
  • "Level of engagement" = % that view the page end up interacting with it (e.g. watching page, posting a reply, starting a new thread)

Measurement ideas (second order effects)

• Increase in the number of "positive/productive" contributions happening on Wikipedia user and article talk ages
• Increase in the % of discussions – started by humans – that are replied to

@ppelberg - Adding some thoughts re potential quantitative metrics below as I think through these. Happy to have a quick meeting if it's easier to figure these out synchronously instead.

Measurement ideas (directly measurable)

• Increase the number of contributions by junior contributors to Wikipedia talk pages (article and user)

Pros: (1) This would be easy to track with current instrumentation and the team can easily monitor changes using the edit_hourly dataset in Turnilo/Superset. (2) This would be the best metric to show overall changes in talk page activity resulting from the team's deployments if that's the primary interest.

Cons: (1) Since we are not differentiating between unique users, this metric will not indicate if the majority of these contributions are coming from a very small group or a large group of junior contributors. (2) Might be more susceptible to noise (could include a lot of minor edits by one user) and hard to isolate changes directly due to deployments.

Looking at the past five years, there’s been a slight overall decline in this metric with an average monthly year over year change of -2.9%; however, there has been a small recent increase in edits on talk pages the past few months. Based on this, we might want to set a smaller target (maybe 3-5%).

• Increase the number of junior contributors making contributions to Wikipedia talk pages (article and user)

Pros: (1) This would be consistent with one of the overall project KRs we set for the project so something we would already be tracking. We could just set a smaller target such as 2-3% to reach by the end of this year. (2) Since we would be tracking distinct users, this metric would provide more insight into the impact of the deployments on the number of contributors but not necessarily overall talk page activity. (3) A little less susceptible to noise compared to looking at overall edit activity.

Cons (2): Not great if we are more interested in looking at overall talk page activity; (3) Data is not currently readily available in Turnilo/Superset but easy enough for me to query.

As shown in T233261, this metric follows similar trends as the number of contributions to talk pages but with a little less fluctuation. There is an average monthly year over year change of -0.86% looking at the past 5 years (Note this is based on all talk page namespaces - not just user and talk)

• Increase the talk page "contribution rate" of junior contributors.
  • "Contribution rate" = % of junior contributors that start to post a reply or new discussion and follow through to post their "thought"

Pros: (1) I like this one as it most directly tracks impact/usage of some of the planned deployments (such as replying to a specific comment)

Cons: (2) we currently don’t have the instrumentation to distinguish a post or reply on a talk page. It would be hard to establish a baseline to identify a realistic target at this point.

• Increase in junior contributors' level of engagement with talk pages
  • "Level of engagement" = % that view the page end up interacting with it (e.g. watching page, posting a reply, starting a new thread)

Pros: This one could be interesting because it helps further isolate the audience to only contributors who actually view the page. Cons (1) More difficult/time-intensive to measure based on current data sources and instrumentation (2) Not sure it would add much more insight into contributor engagement on talk pages compared to looking at the change in the number of junior contributors who edit a talk page.

@ppelberg: Could you please clarify / fix the "FY2020" which does not exist? Thanks!

In T234409#5564801, @MNeisler wrote:

@ppelberg - Adding some thoughts re potential quantitative metrics below as I think through these. Happy to have a quick meeting if it's easier to figure these out synchronously instead..

The thinking @MNeisler shared in the above became the basis for the conversation we had yesterday.

During yesterday's conversation, we came to think that our goal for this year (30-Sep-2019 – 30-June-2020) should be to: Increase the number of Junior Contributors [1] making productive contributions to Wikipedia's talk pages, across these 16 talk namespaces.

We thought the metric above best met these criteria:

• A metric we have enough data to calculate a baseline for
• A metric we think the interventions we deploy this year [2] can directly affect
• A metric that relates/serves our project objectives T233888

To be clear, this is not the only metric we will be measuring this year...we will measure the impact of the incremental improvements we release. Although, the metric we are defining in this task will be the metric we will use to evaluate the sum impact of our work over this next year.

1. Metric definitions can be found here: T234046
2. This year, we are intending to release interventions that make:
   • Contributing content to talk pages more efficient and inuitive
   • Understanding and following the activity on talk pages easier
   • Discovering talk pages easier

Adding a few notes from a conversation with @kzimmerman just now...

• Next year, when we're looking at the impact our work has had on the number of Junior Contributors making productive contributions to Wikipedia talk pages, we should be sure to also check how this work has impacted the number of Junior Contributors making productive contributions to Wikipedia articles.
  • Thinking: we wouldn't want our work to improve talk pages to degrade contributions to Wikipedia content pages