Page MenuHomePhabricator

Topic subscriptions: calculate engagement metrics
Closed, DeclinedPublic


This task is about analyzing how people are engaging with the Topic Subscription feature while it's been made available as an opt-out preference at a sub-set of wikis to help us determine whether it is ready to be made available as an opt-out preference at more wikis.

Analysis timing




  • Learn whether Junior Contributors are able to use Topic Subscriptions successfully.
  • Learn whether Topic Subscriptions are negatively affecting other volunteers' experiences.

Decision(s) to be made

  • Decide whether any changes need to be made to the notification interventions before measuring the impact they have on peoples' likelihood to receive more timely responses to the things they say on Wikipedia talk pages.

Research questions

  1. Are Junior and Senior Contributors able to use manual topic subscriptions successfully?
  2. Are Junior and Senior Contributors finding topic subscriptions valuable?
  3. Are Topic Subscriptions negatively impacting peoples' engagement with talk pages? E.g. are we seeing a sudden increase in the percentage of new comments that are reverted within 48 hours of being posted? Are we seeing a sudden increase in the percentage of people who are blocked after posting a new comment on a talk page? Are we seeing a sudden decrease in the percentage of notification that people do NOT open, by type? Are we seeing a sudden increase in the number of Echo notifications active editors are receiving per day?

To help answer the questions above, we would like to understand the "Engagement metrics" listed below.

Engagement metrics

@MNeisler to draft.



  • The ===Engagement metrics are documented in the section above
  • The ===Engagement metrics are calculated and a link to these metrics is shared in this ticket

Event Timeline

ppelberg moved this task from Backlog to Triaged on the DiscussionTools board.
ppelberg moved this task from Untriaged to Next Quarter on the Editing-team board.
ppelberg updated the task description. (Show Details)
ppelberg added a subscriber: MNeisler.

Per the conversation @MNeisler and I had offline on 2 Feb, we are going to use this analysis as a means to decide whether we think Topic Subscriptions are in a state where an A/B test can be started to evaluate the impact they are having on peoples' likelihood to participate in talk page discussions.

The primary implication of the above is foregoing work on T280898.

Note: what we are seeking to learn through this analysis remains unchanged: Learn the extent to which Senior Contributors are trying Topic Subscriptions (Manual and Automatic) and finding them valuable.