Superset: extend DiscussionTools dashboard to include topic subscription metrics
Closed, ResolvedPublic
Actions

Description

This task involves the work with extending the existing DiscussionTools Superset dashboard @MNeisler created [i] to include metrics about high-level topic subscription (T263820 + T263819) usage.

Objectives

The metrics described in the ===Requirements section below are intended to help the Editing Team identify unexpected trends in usage of Manual (T263820) and Automatic Topic Subscriptions (T263819) that will serve as a signal that further investigation is needed.

Requirements

ADD the following two charts to the existing DiscussionTools Superset dashboard [i]:
- Chart 1: a graph showing the total number of topic subscriptions people have initiated to date, filterable by wiki and how the topic subscription was initiated (manually or automatically).
- Chart 2: a graph showing the total number of topic subscriptions that are currently ACTIVE, filterable by wiki and how the topic subscription was initiated (manually or automatically).
"Chart 1" and "Chart 2" should be implemented such that they are populated with real-time data.
- Note: if this requirement surpasses Superset's data limits, we will adjust the charts such that they are updated at a slower cadence (e.g. weekly, bi-weekly, monthly, etc.).

Done

The DiscussionTools Superset dashboard [i] is updated to include the two charts described in the ===Requirements section above

i. https://superset.wikimedia.org/r/632

Related Objects
Search...

Status	Assigned	Task
Open	None	T276497 Scale DiscussionTools to all projects
Open	None	T251207 [Epic] Scale DiscussionTools to all Wikipedias
Resolved	• ppelberg	T284489 Deploy topic subscriptions (desktop) as opt-out feature at all projects
Resolved	• ppelberg	T284490 Offer topic subscriptions as a Beta Feature at all projects
Resolved	• ppelberg	T287277 Offer automatic topic subscriptions as Beta Feature at all projects
Open	None	T233447 [OKR] Increase response rates
Open	None	T273920 [Interventions] Notifications
Duplicate	• ppelberg	T273906 [Experiment ] Topic Subscriptions
Open	None	T273921 Notifications: Research
Resolved	• ppelberg	T273908 Notifications: Research: Identify usability control test findings for Notifications
Resolved	• iamjessklein	T273909 Notifications: Research: Audit notification types
Open	MNeisler	T259184 How are notifications received
Open	MNeisler	T249762 Analyze what impacts response rate
Resolved	• ppelberg	T258816 Develop an approach for talk page notifications
Declined	None	T258817 [Product Investigation] What do we currently know about the impact of notifications?
Open	None	T262104 [SPIKE] How many accounts have email addresses associated with them?
Open	None	T264463 [RESEARCH SPIKE] How do Senior Contributors keep track of conversations on other users' talk pages?
Open	None	T263819 [Intervention] Automatic topic subscriptions
Resolved	• ppelberg	T282636 [Release Ticket] Offer Automatic Topic Subscriptions as Beta Feature at Initial Wikis
Resolved	Esanders	T263821 [Intervention] Introduce New Topic Notifications (desktop)
Open	None	T263820 [Intervention] Manual topic subscriptions
Resolved	• LZaman	T274188 Offer manual topic subscriptions as a Beta Feature at partner wikis
Resolved	• iamjessklein	T273907 Topic Subscriptions: Design Brief
Duplicate	None	T260371 Develop an approach for storing thread subscriptions
Resolved	JTannerWMF	T260372 [SPIKE] What is the process for creating a table to store thread subsriptions?
Resolved	Esanders	T272197 Update Discussion tools beta feature to include topic subscriptions
Resolved	• ppelberg	T274193 Identify topic subscription "partner wikis"
Resolved	• ppelberg	T274201 Consult with Senior Contributors about topic subscription requirements
Resolved	• iamjessklein	T266203 Notifications: Research: Map Junior Contributors' current notification experience
Resolved	MNeisler	T274215 Create response rate measurement plan
Resolved	• ppelberg	T274216 Build topic subscriptions v1.0
Resolved	matmarex	T264478 Changes to DiscussionTools internals to better support watching/notifications
Resolved	matmarex	T263817 DBA review: conversation subscriptions
Resolved	Esanders	T264885 Investigate table schema / DB abstraction layer for section/comment watching/unwatching
Resolved	matmarex	T274685 Comment IDs that reliably identify the comment across revisions/pages (comment names)
Resolved	matmarex	T275881 Don't search for signatures in blockquote elements
Resolved	Ryasmeen	T275949 Implement topic subscription notification bundling
Resolved	Esanders	T276990 Ensure duplicate notifications are not sent for the same talk page event
Resolved	• ppelberg	T279671 Ensure that all of topic subscriptions code is behind the feature flag
Resolved	Esanders	T280783 Only fire notifications for new comments when signature matches current user
Resolved	matmarex	T281000 Improve query for fetching user's subscriptions on a page
Resolved	Urbanecm	T282699 Create topic subscription table
Duplicate	None	T274217 Topic subscriptions v1.0 usability testing (Senior Contributors)
Resolved	• ppelberg	T274218 Publish topic subscriptions usability test findings on mediawiki.org
Duplicate	None	T274278 Finalize topic subscription QQQ messages
Resolved	Ryasmeen	T274279 Pre-deployment regression testing (topic subscriptions)
Resolved	matmarex	T274280 Deploy config change to make Topic Subscriptions available as beta feature
Resolved	• Whatamidoing-WMF	T274281 Announce Topic Subscriptions availability as beta feature
Resolved	• ppelberg	T275232 Learn what Senior Contributors think about topic subscription initial prototype
Resolved	• iamjessklein	T273911 Topic Subscriptions: Prototype Workflow
Resolved	matmarex	T279119 Investigate why comments are slow to post on patchdemo
Resolved	matmarex	T279395 Unable to create a new account with patch demo
Resolved	• ppelberg	T276996 Hide topic subscription affordance on peoples' own user talk page
Resolved	• iamjessklein	T279164 Topic subscription: validate usability of topic subscription prototype with Juniors
Resolved	• iamjessklein	T273912 Topic Subscriptions: Design usability test (Junior Contributors)
Resolved	Esanders	T279149 Topic Subscriptions: Improve user interface element for subscribing and unsubscribing to topics on Talk Pages
Resolved	Esanders	T279150 Topic Subscriptions: Unsubscribe from Notifications from within dropdown
Resolved	• ppelberg	T279151 Topic Subscriptions: Define copy and ensure consistent treatment across touchpoints for Notifications
Resolved	• ppelberg	T279498 [SPIKE] Determine how Topic Subscriptions and Watchlists relate in the near-term
Resolved	matmarex	T280199 Implement a way for test comments to be posted automatically
Resolved	• ppelberg	T281438 Topic Subscriptions: Run manual topic subscription usability test with Junior Contributors
Resolved	matmarex	T281471 Topic Subscription Prototype: Highlight thread after following a notification
Resolved	Ryasmeen	T284120 Post-deployment QA (Manual Topic Subscriptions)
Resolved	matmarex	T284169 Deploy config change to make Topic Subscriptions available on the beta cluster
Resolved	matmarex	T284175 Topic subscription notifications are not sent on the beta cluster
Resolved	• ppelberg	T262990 [Technical investigation] What happens if a specific section you are watching gets moved?
Resolved	• ppelberg	T262991 [Technical investigation] What happens if a section you are watching gets renamed?
Open	None	T275943 Add support for subsection subscriptions
Duplicate	None	T280411 Offer topic subscriptions as opt-out feature at partner wikis
Invalid	None	T280412 [SPIKE] Develop approach for optimizing topic subscription storage
Resolved	Esanders	T332730 [Intervention] Introduce New Topic Notifications (mobile)
Duplicate	MNeisler	T280895 Evaluate impact of notification interventions
Resolved	• ppelberg	T287126 Superset: extend DiscussionTools dashboard to include topic subscription metrics

Event Timeline

• ppelberg created this task.Jul 21 2021, 9:41 PM

• ppelberg moved this task from Backlog to Analytics on the Editing-team (Tracking) board.

• ppelberg moved this task from Backlog to Triaged on the DiscussionTools board.

• ppelberg added a parent task: T233447: [OKR] Increase response rates.

• ppelberg edited parent tasks, added: T280895: Evaluate impact of notification interventions; removed: T233447: [OKR] Increase response rates.

MNeisler triaged this task as Medium priority.Jul 22 2021, 3:07 PM

MNeisler moved this task from Triage to Upcoming Quarter on the Product-Analytics board.

MNeisler claimed this task.Jul 27 2021, 2:40 PM

• ppelberg added a parent task: T282636: [Release Ticket] Offer Automatic Topic Subscriptions as Beta Feature at Initial Wikis.Aug 26 2021, 4:55 PM

MNeisler edited projects, added Product-Analytics (Kanban); removed Product-Analytics.Aug 26 2021, 5:23 PM

MNeisler moved this task from Next 2 weeks to Doing on the Product-Analytics (Kanban) board.

• LZaman mentioned this in T288480: [Release ticket] Offer Manual Topic Subscriptions as Beta Feature at Phase 3 Wikis.Aug 26 2021, 9:11 PM

• LZaman added a project: Editing QA.

• LZaman mentioned this in T282636: [Release Ticket] Offer Automatic Topic Subscriptions as Beta Feature at Initial Wikis.Aug 26 2021, 9:23 PM

I'd like to know how many sections have been subscribed to at enwiki (not necessarily counting mine, because I'm using it a lot). We had an active discussion about this starting last Friday (27 August 2021), so I think that 26 August is the baseline state. I'm curious whether this discussion will result in much wider use on other pages.

@ppelberg I've updated the DiscussionTools Dashboard (currently renamed to Talk Pages Project Dashboard) to include the requested charts on topic subscription data. Data comes reflects a current snapshot of topic subscriptions as of 31 August 2021.

I'd like to do a little more QA of the data so keeping this assigned to me right now but please let me know if you have any questions or suggested changes to the proposed charts. See details about the charts and details below:

Chart Descriptions: For both initiated and active topic subscriptions, I've currently included charts that show the overall totals as of 31 August 2021 and daily totals by the date the subscription was created. Per @Whatamidoing-WMF's suggestion in T287126#7319535, I also included a chart to show the number of distinct sections/topics have been subscribed to. All charts are filterable by wiki using the provided Topic Subscriptions Filter on the dashboard.
Filters: All charts are filterable by wiki using the provided Topic Subscriptions Filter on the dashboard. There is also an 'initiation type' filter to view data by how the topic subscription was initiated (manually or automatically). However, data currently only reflects manual topic subscriptions pending implementation of automatic topic subscriptions.
Frequency of Updates These charts are unable to be updated in real-time as the data source for these charts comes from a query that currently needs to be manually run and then uploaded into Superset. This process does not take long but will require some planning and discussion on the needed frequency of updates since it's not automated. If needed frequently, a job scheduler can eventually be set-up to automate this process.
Data comes from the discussiontools_subscription table, which stores a creation timestamp and state of subscription but does not track changes in user preferences over time (for example, if a user is currently unsubscribed to a topic, we will not be able to determine if that user was subscribed manually or automatically). Implications for these charts: Only the active topic subscriptions charts are filterable by initiation type.

• ppelberg removed a project: Editing QA.Sep 2 2021, 7:18 PM

@ppelberg - I completed QA and confirmed that the data in Topic Subscriptions charts is correct. Moving to you for review and final sign-off. Please let me know if you have any questions or suggested changes.

• ppelberg mentioned this in T290516: Enable topic subscription dashboard to update regularly .Sep 7 2021, 6:36 PM

In T287126#7324724, @MNeisler wrote:

@ppelberg I've updated the DiscussionTools Dashboard (currently renamed to Talk Pages Project Dashboard) to include the requested charts on topic subscription data. Data comes reflects a current snapshot of topic subscriptions as of 31 August 2021.

Chart Descriptions: For both initiated and active topic subscriptions, I've currently included charts that show the overall totals as of 31 August 2021 and daily totals by the date the subscription was created. Per @Whatamidoing-WMF's suggestion in T287126#7319535, I also included a chart to show the number of distinct sections/topics have been subscribed to. All charts are filterable by wiki using the provided Topic Subscriptions Filter on the dashboard.

This looks GREAT, @MNeisler!

To be doubly sure I am understanding these charts accurately, can you please let me know what – if anything – about the below is inaccurate?

Initiated topic subscriptions (total): a count of the total number of times people have subscribed to a topic. For right now, this chart is limited to manual topic subscriptions. Eventually, it will also include automatic topic subscriptions.
- E.g. if Person A and Person B both subscribe to Topic 1 on 1-September, the chart will show a count of "2 initiated topic subscriptions" on this day.
Active topic subscriptions by date created: the total number of times people have subscribed to a topic minus the total number of times people have unsubscribed from topics.
- E.g. if Person A and Person B both subscribe to Topic 1 on 1-September, and then Person A unsubscribes from Topic 1, the chart will show a count of "1 active topic subscriptions" on this day.
Distinct topics subscribed to (total): a count of the distinct conversations at least one person has subscribed to.
- E.g. if Person A and Person B both subscribe to Topic 1 on 1-September, the chart will show a count of "1 distinct topics subscribed to" on this day.

Filters: All charts are filterable by wiki using the provided Topic Subscriptions Filter on the dashboard. There is also an 'initiation type' filter to view data by how the topic subscription was initiated (manually or automatically). However, data currently only reflects manual topic subscriptions pending implementation of automatic topic subscriptions.

Noted.

Frequency of Updates These charts are unable to be updated in real-time as the data source for these charts comes from a query that currently needs to be manually run and then uploaded into Superset. This process does not take long but will require some planning and discussion on the needed frequency of updates since it's not automated. If needed frequently, a job scheduler can eventually be set-up to automate this process.

Understood. The follow up work needed to ensure the topic subscription data updates in real-time is now being tracked in this newly-filed ticket: T290516.

Data comes from the discussiontools_subscription table, which stores a creation timestamp and state of subscription but does not track changes in user preferences over time (for example, if a user is currently unsubscribed to a topic, we will not be able to determine if that user was subscribed manually or automatically). Implications for these charts: Only the active topic subscriptions charts are filterable by initiation type.

Understood.

@ppelberg

In T287126#7337399, @ppelberg wrote:

To be doubly sure I am understanding these charts accurately, can you please let me know what – if anything – about the below is inaccurate?

Initiated topic subscriptions (total): a count of the total number of times people have subscribed to a topic. For right now, this chart is limited to manual topic subscriptions. Eventually, it will also include automatic topic subscriptions.

E.g. if Person A and Person B both subscribe to Topic 1 on 1-September, the chart will show a count of "2 initiated topic subscriptions" on this day.

Active topic subscriptions by date created: the total number of times people have subscribed to a topic minus the total number of times people have unsubscribed from topics.

E.g. if Person A and Person B both subscribe to Topic 1 on 1-September, and then Person A unsubscribes from Topic 1, the chart will show a count of "1 active topic subscriptions" on this day.

For these two statements above, it's not completely accurate to state "the total number of times people have subscribed or unsubscribed from a topic". We currently only keep data on the current state of each person's subscription and do not know if that person changed their subscriptions status for that topic multiple times. For example, so if Person A unsubscribes from Topic 1 and then decides to resubscribe to Topic 1 later and Person B stayed subscribed the whole time, both of these subscriptions states would be reflected as "2 active subscriptions" in the data. Also, in this scenario, Person A subscribing to Topic 1 would be counted as just 1 initiated topic subscription even though they technically subscribed to the same topic multiple times.

It would be more accurate to say the following:
Initiated topic subscriptions (total): "the total number of times people first subscribed to a topic" or "the total number of times people initiated a topic subscription"
Active topic subscriptions by date created: "the total number of times people first subscribed to a topic minus the current total of topic subscriptions that are currently set as unsubscribed "

Distinct topics subscribed to (total): a count of the distinct conversations at least one person has subscribed to.

E.g. if Person A and Person B both subscribe to Topic 1 on 1-September, the chart will show a count of "1 distinct topics subscribed to" on this day.

Correct.

How to interpret the Line Graphs
Also, a quick note on how to read the dates in the line charts. For all line charts, the dates on the x-axis reflect the date the subscription was first created/initiated since the data is based on a snapshot and we don't currently record changes in an initiated topic subscription over time. This can make interpreting the "Active topic subscriptions by date created" slightly more confusing to read (If this is more confusing than beneficial, we might want to consider removing this chart and just keeping the "Active topic subscriptions (total)" chart instead).

I've provided examples on how to read each chart below for the date of July 10th:

"Initiated topic subscriptions by date created": There were 25 topic subscriptions created on July 10th.
"Active topic subscriptions by date created": 24 of the topic subscriptions created on July 10th are still active as of 31 August 2021.
"Distinct topics subscribed to by date created": There were 25 distinct topics that were subscribed to on July 10th.

Per the meeting @MNeisler and I had on 9 September, all that's left to be done on this task is for Megan to update the Talk Pages Project > Topic Notifications dashboard with the clarifying language she shared in T287126#7342081.

• ppelberg mentioned this in T291195: Add MariaDB replicas to Superset.Sep 16 2021, 7:41 PM

In T287126#7349641, @ppelberg wrote:

Per the meeting @MNeisler and I had on 9 September, all that's left to be done on this task is for Megan to update the Talk Pages Project > Topic Notifications dashboard with the clarifying language she shared in T287126#7342081.

@ppelberg
This is complete. I also went ahead and updated the topic subscriptions dashboard so it reflects data as of 16 September 2021.
Please let me know if you have any questions or additional modifications to complete as part of this task.

• ppelberg claimed this task.Sep 18 2021, 1:16 AM

• ppelberg edited projects, added Editing-team (Kanban Board); removed Editing-team (Tracking).

• ppelberg moved this task from Incoming to Ready for Sign Off on the Editing-team (Kanban Board) board.

• ppelberg closed this task as Resolved.Sep 22 2021, 3:55 PM

• ppelberg mentioned this in T291872: Adjust "Active topic subscriptions" chart in Superset dashboard.Sep 27 2021, 7:18 PM