Note: I've updated the task description with the metrics @MNeisler converged on during the meeting we had on 17 November.

Meta
Per the conversation @MNeisler and I had offline on 2 Feb, we are going to use this analysis as a means to decide whether we think Topic Subscriptions are in a state where an A/B test can be started to evaluate the impact they are having on peoples' likelihood to participate in talk page discussions.

The primary implication of the above is foregoing work on T280898.

Note: what we are seeking to learn through this analysis remains unchanged: Learn the extent to which Senior Contributors are trying Topic Subscriptions (Manual and Automatic) and finding them valuable.

@ppelberg
Below are the key findings from the topic notification adoption metric analysis for review.
Please see the notebook for further details and let me know if you have any questions.

1. Are people aware of and trying the topic subscription feature?
Metric: Of the people who have published an unreverted talk page edit in the time since topic notification became available, what percent have subscribed to at least one topic. Note: we cannot know whether the topic someone subscribed to is a topic they also edited/contributed to.

• 28% of all users (40% of Senior Contributors) that edited a talk page and have the discussiontools-betaenable beta preference enabled have subscribed to at least 1 topic since 19 October 2021 (when manual topic notifications was available as a beta feature to all Wikis).

2. Are people finding the topic subscription feature valuable?
Metrics:
[Manual] Of the people who have manual topic subscriptions enabled, what percentage of people are actively subscribed to the following number of discussions?

The majority (65%) of users actively subscribed to manual topic subscriptions are subscribed to 1 to 2 topics. 35% of manual topic subscribers have subscribed to over 2 distinct topics.

[Automatic] Of the people who have manual topic subscriptions enabled (and edited a talk page [i]), what percentage of them enabled automatic topic subscriptions within Special:Preferences?

• There are 537 distinct users that have enabled the automatic topic subscription preference across all wikis where available.
• 6% of all users (6.4% of Senior Contributors) that have manual topic subscription enabled and edited a talk page since the deployment of automatic topic subscription feature[ii])

i. I limited the analysis to users that edited a talk page since deployment of the automatic topic preference so we can get a better estimate of users that have seen the feature and enabled it. There may be many users that have manual topic subscriptions enabled but have not visited a talk page since the deployment of the new feature as it's only been available for a couple of months.
ii. This excludes wikis where the automatic topic subscription feature is not yet available.

[Both] Percentage of new comment notifications that Junior and Senior Contributors engage with (e.g. make as read, tap a notification) within two weeks of receiving them.

• 96% (92.5%) of topic notifications sent to topic subscribers (including both manual or automatic topic subscribers) are read within 2 weeks of receiving them. Here is the split between automatic and manual topic subscribers:
  • 98.4% of notifications sent to automatic topic subscribers are read within 2 weeks
  • 92.3% of notifications sent to manual topic subscribers are read within 2 weeks

[i]With currently available instrumentation, we do not know the type of notification that was sent and/or read. As a result, I limited this analysis to contributors that have subscribed to a topic (either automatic or manual). Note: This may still include some notifications received by topic subscribers that were not sent from the topic notifications feature. (Instrumentation confirmed in T280896#7757798)

Are people finding the topic subscription feature disruptive?
[Automatic] Of the people who enabled Automatic Topic Subscriptions, what percentage of people disable the feature after having been automatically subscribed to at least one conversation?

• 23% of users (61 of 259 users that were automatically subscribed to a topic) disabled the automatic topic subscription preference in their user properties after being automatically subscribed to a topic.

[Both] Sudden increase in the average number of notifications sent per person, per day.

The average number of notifications sent per day has remained fairly stable with a daily average of about 4 notifications per user per day. There have been some slight increases in the average daily number of topic notifications following the release of manual topic subscriptions at all Wikipedias on 29 August 2021 and at all Wikis on 19 October 2021. This may be indicative of it being available to more active users of the feature.

A couple other observations:

• There was on spike on 1 November 2021 to 16 topic notifications per user day but this increase was not sustained and there have been no further spikes since that date.
• The deployment of automatic topic subscriptions as a beta feature did not significantly increase the average number of topic notifications per day. Since this feature has currently only been enabled by a small number of users, it would be worth continuing to monitor the impacts of this feature on the rate of notifications as it is more widely deployed.

[Both] Of the contributors that receive a notification, what percent turn them off?

Manual Topic Subscriptions: 0.5% of all users that received a topic notification explicitly disabled the manual topic subscription preference.
Automatic Topic Subscription: 18% of automatic topic notification users that received a topic notification explicitly disabled the automatic topic preference.

I took a closer review of active topic subscriptions by the number of subscriptions to better understand what "power usage" of this beta feature looks like.

Observations:

• Of the users that have manually subscribed to a topic, the majority of users (65%) have subscribed to 1 to 2 distinct topics on talk pages. The percent of topic subscribers gradually decreases as the number of topic subscriptions increases. However, we also found that there are very active users of the feature, that subscribe to over 20 topic subscriptions.
• There are 130 distinct users (6% of all active manual topic subscribers) that are actively subscribed to over 20 topic subscriptions.

Due to this highly skewed nature of the data, I reviewed the usage of the topic subscriptions features by users with over 20 topic subscriptions on a separate chart. This helps us to better visualize and understand "power usage" of the manual topic subscription feature.

• Of these "power users", most (55%) subscribe to between 21 to up to 40 topic subscriptions. Power users are almost as likely to subscribe to 31 to 40 topic subscriptions as they are to subscribe to between 21 to 30.
• After 40 topic subscriptions, the percent of users drops significantly and generally continues to decrease with a small percentage (13%) of users subscribing to over 100 topic subscriptions.

@ppelberg
Please let me know if that helps clarify or if there are any additional breakdowns or summaries that would be useful in understanding the usage of this tool.

[i]With currently available instrumentation, we do not know the type of notification that was sent and/or read. As a result, I limited this analysis to contributors that have subscribed to a topic (either automatic or manual). Note: This may still include some notifications received by topic subscribers that were not sent from the topic notifications feature.

Determined in T303205 that instrumentation is available in the echo_event table to determine event_type.

Since it was a quick change, I went ahead and updated the results for any of the "notification-sent based metrics" based on this instrumentation. I've also updated these numbers in the report and report summary provided in T280896#7749608.

In T280896#7757407, @MNeisler wrote:

Please let me know if that helps clarify or if there are any additional breakdowns or summaries that would be useful in understanding the usage of this tool.

@MNeisler: per the discussion we had on 9 March offline, the results look great.

I've published them on-wiki in this edit: https://w.wiki/4wYj.