Page MenuHomePhabricator
Create Task
Maniphest T286366

Implement product key performance indicator monitoring for Growth features in Grafana
Closed, ResolvedPublic
Actions

Description

This task follows up a retro discussion of T285577: Several wikis have 0 articles for all ORES topics, where articletopic stopped returning results, and we found out about it from a community member alerting us to this. T286365: Implement monitoring for articletopic / weighted_tags exists for monitoring specific to that field, but this task is more specific to instrumentation we want to add in GrowthExperiments and log to statsd so that it can appear in a Grafana dashboard (possibly our existing one), and we'd be able to see earlier on if there is a problem.

Let's first list out which things we want to output in statsd and then think about whether they can be aggregated or need to be separated by wiki (and we can make subtasks for them if needed):

  • newcomer task edit count
  • structured task newcomer task edit count
  • newcomer task count split by task type ID
  • accounts created that are opted into growth features
  • questions asked to mentor from help panel
  • questions asked to mentor from special:homepage
  • clicks on tasks, segmented by task type

Details

SubjectRepoBranchLines +/-
Monitoring: Add '.Save' to distinguish from '.Click' eventsmediawiki/extensions/GrowthExperimentswmf/1.38.0-wmf.18+3 -1
Monitoring: Add '.Save' to distinguish from '.Click' eventsmediawiki/extensions/GrowthExperimentswmf/1.38.0-wmf.17+3 -1
Monitoring: Add '.Save' to distinguish from '.Click' eventsmediawiki/extensions/GrowthExperimentsmaster+3 -1
Monitoring: Adjust logic for counting revertsmediawiki/extensions/GrowthExperimentswmf/1.38.0-wmf.16+10 -5
Monitoring: Adjust logic for counting revertsmediawiki/extensions/GrowthExperimentsmaster+10 -5
Monitoring: Track task click rate by task type IDmediawiki/extensions/GrowthExperimentsmaster+12 -4
Monitoring: Track question asking for mentorshipmediawiki/extensions/GrowthExperimentsmaster+39 -3
Monitoring: Track revert rate for newcomer tasksmediawiki/extensions/GrowthExperimentsmaster+34 -1
Monitoring: Track users opted into growth features and variantsmediawiki/extensions/GrowthExperimentsmaster+4 -0
Monitoring: Track newcomer task saves by task type, per wikimediawiki/extensions/GrowthExperimentsmaster+10 -0
Customize query in gerrit

Related Objects
Search...

Event Timeline

kostajh created this task.Jul 9 2021, 7:59 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 9 2021, 7:59 AM
kostajh moved this task from Inbox to Needs Discussion on the Growth-Team board.Jul 9 2021, 7:59 AM
kostajh added a comment.Jul 9 2021, 12:26 PM

There's certainly some overlap with T249987: Scale: GrowthExperiments wiki monitoring dashboard, but I feel like that one has a different scope and frame of reference (more holisitic, longer term / macro view of the features), while this is about knowing early on when something is broken.

kostajh triaged this task as Low priority.Oct 5 2021, 9:18 AM
kostajh edited projects, added Growth-Team (Sprint 0 (Growth Team)); removed Growth-Team.

Moving into current sprint per team discussion.

kostajh mentioned this in T294755: Projects for interim period.Nov 24 2021, 2:13 PM
Trizek-WMF subscribed.Dec 13 2021, 6:26 PM
kostajh updated the task description. (Show Details)Dec 13 2021, 6:27 PM
kostajh moved this task from Incoming to Ready for Development on the Growth-Team (Sprint 0 (Growth Team)) board.Dec 13 2021, 6:33 PM

This doesn't need to all get done at once. We can split these out into subtasks if desired.

kostajh raised the priority of this task from Low to Medium.Dec 13 2021, 6:50 PM
Trizek-WMF awarded a token.Dec 13 2021, 7:36 PM

Looking forward to see some ways to monitor being built, so that I can provide data to communities in an easy way!

gerritbot added a comment.Dec 13 2021, 9:13 PM

Change 746955 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):

[mediawiki/extensions/GrowthExperiments@master] Monitoring: Track newcomer task saves by task type, per wiki

https://gerrit.wikimedia.org/r/746955

gerritbot added a project: Patch-For-Review.Dec 13 2021, 9:13 PM
gerritbot added a comment.Dec 13 2021, 9:17 PM

Change 746956 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):

[mediawiki/extensions/GrowthExperiments@master] Monitoring: Track users opted into growth features and variants

https://gerrit.wikimedia.org/r/746956

Tgr added a project: GrowthExperiments.Dec 13 2021, 11:24 PM
gerritbot added a comment.Dec 14 2021, 2:30 PM

Change 747126 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):

[mediawiki/extensions/GrowthExperiments@master] Monitoring: Track revert rate for newcomer tasks

https://gerrit.wikimedia.org/r/747126

kostajh claimed this task.Dec 14 2021, 2:30 PM
kostajh moved this task from Ready for Development to Code Review on the Growth-Team (Sprint 0 (Growth Team)) board.
kostajh mentioned this in T297709: Mentorship / help panel tags not used with structured discussion talk pages.Dec 14 2021, 3:44 PM
gerritbot added a comment.Dec 14 2021, 4:05 PM

Change 747152 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):

[mediawiki/extensions/GrowthExperiments@master] Monitoring: Track question asking for mentorship

https://gerrit.wikimedia.org/r/747152

kostajh added a comment.Dec 14 2021, 4:14 PM

I'm unsure about these:

  • unstructured task pool size (no topics)
  • unstructured task pool size (with topics)

The rest are ready for review.

kostajh renamed this task from Implement monitoring for Growth features to Implement product key performance indicator monitoring for Growth features in Grafana.Dec 15 2021, 8:46 AM
kostajh added subscribers: DMburugu, nettrom_WMF, MMiller_WMF, RHo.Dec 15 2021, 8:56 AM

I've added a Product KPIs chore table to https://www.mediawiki.org/wiki/Growth/Team/Chores and my proposal is that it would work like this:

  • engineers review the sections on the new Grafana page for product KPIs, and write in the "Notes" column of the Product KPIs table if there's something that seems off
  • folks who are not involved in the engineer chores are the ones who are responsible for updating the status and signing their name that they've reviewed the chart. This would be a task to be done once a week and would probably take 5-15 minutes of time – you would glance at the charts, and note anything that seems problematic. Provisionally that is @DMburugu, @MMiller_WMF / new PM, @RHo, @nettrom_WMF and @Trizek-WMF but we could discuss if that is something you all would rather not do, or if we think it should only be 1-2 people, or less frequently, etc.

Once the Grafana page is set up with monitoring metrics, we can continue the discussion about who/how we monitor it.

gerritbot added a comment.Dec 15 2021, 2:31 PM

Change 747519 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):

[mediawiki/extensions/GrowthExperiments@master] Monitoring: Track task click rate by task type ID

https://gerrit.wikimedia.org/r/747519

kostajh updated the task description. (Show Details)Dec 15 2021, 2:35 PM
Trizek-WMF added a comment.Dec 15 2021, 6:12 PM

I personally would prefer data separated by wiki, since I time to time have requests from some wikis about data regarding efficiency.

Is it possible to get the revert rate by type of tasks?

kostajh added a comment.Dec 15 2021, 7:39 PM
In T286366#7573199, @Trizek-WMF wrote:

I personally would prefer data separated by wiki, since I time to time have requests from some wikis about data regarding efficiency.

Yes, all the metrics here are per wiki, though we will probably also want some panel that aggregates data.

Is it possible to get the revert rate by type of tasks?

Yes, but after T297004: Rework how we track newcomer task edits is done.

gerritbot added a comment.Dec 18 2021, 8:04 AM

Change 746955 merged by jenkins-bot:

[mediawiki/extensions/GrowthExperiments@master] Monitoring: Track newcomer task saves by task type, per wiki

https://gerrit.wikimedia.org/r/746955

ReleaseTaggerBot added a project: MW-1.38-notes (1.38.0-wmf.16; 2022-01-03).Dec 18 2021, 9:00 AM
gerritbot added a comment.Jan 4 2022, 12:31 AM

Change 746956 merged by jenkins-bot:

[mediawiki/extensions/GrowthExperiments@master] Monitoring: Track users opted into growth features and variants

https://gerrit.wikimedia.org/r/746956

gerritbot added a comment.Jan 4 2022, 12:47 AM

Change 747126 merged by jenkins-bot:

[mediawiki/extensions/GrowthExperiments@master] Monitoring: Track revert rate for newcomer tasks

https://gerrit.wikimedia.org/r/747126

gerritbot added a comment.Jan 4 2022, 9:05 AM

Change 751388 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):

[mediawiki/extensions/GrowthExperiments@master] Monitoring: Adjust logic for counting reverts

https://gerrit.wikimedia.org/r/751388

gerritbot added a comment.Jan 4 2022, 9:10 AM

Change 747152 merged by jenkins-bot:

[mediawiki/extensions/GrowthExperiments@master] Monitoring: Track question asking for mentorship

https://gerrit.wikimedia.org/r/747152

gerritbot added a comment.Jan 4 2022, 10:26 AM

Change 747519 merged by jenkins-bot:

[mediawiki/extensions/GrowthExperiments@master] Monitoring: Track task click rate by task type ID

https://gerrit.wikimedia.org/r/747519

gerritbot added a comment.Jan 4 2022, 10:39 AM

Change 751191 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):

[mediawiki/extensions/GrowthExperiments@wmf/1.38.0-wmf.16] Monitoring: Adjust logic for counting reverts

https://gerrit.wikimedia.org/r/751191

gerritbot added a comment.Jan 4 2022, 10:59 AM

Change 751388 merged by jenkins-bot:

[mediawiki/extensions/GrowthExperiments@master] Monitoring: Adjust logic for counting reverts

https://gerrit.wikimedia.org/r/751388

gerritbot added a comment.Jan 4 2022, 12:26 PM

Change 751191 merged by jenkins-bot:

[mediawiki/extensions/GrowthExperiments@wmf/1.38.0-wmf.16] Monitoring: Adjust logic for counting reverts

https://gerrit.wikimedia.org/r/751191

ReleaseTaggerBot edited projects, added MW-1.38-notes (1.38.0-wmf.17; 2022-01-10); removed MW-1.38-notes (1.38.0-wmf.16; 2022-01-03).Jan 4 2022, 4:00 PM
gerritbot added a comment.Jan 18 2022, 11:38 AM

Change 754893 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):

[mediawiki/extensions/GrowthExperiments@master] Monitoring: Add '.Save' to distinguish from '.Click' events

https://gerrit.wikimedia.org/r/754893

gerritbot added a comment.Jan 18 2022, 11:50 AM

Change 754605 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):

[mediawiki/extensions/GrowthExperiments@wmf/1.38.0-wmf.17] Monitoring: Add '.Save' to distinguish from '.Click' events

https://gerrit.wikimedia.org/r/754605

gerritbot added a comment.Jan 18 2022, 11:53 AM

Change 754906 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):

[mediawiki/extensions/GrowthExperiments@wmf/1.38.0-wmf.18] Monitoring: Add '.Save' to distinguish from '.Click' events

https://gerrit.wikimedia.org/r/754906

gerritbot added a comment.Jan 18 2022, 12:55 PM

Change 754893 merged by jenkins-bot:

[mediawiki/extensions/GrowthExperiments@master] Monitoring: Add '.Save' to distinguish from '.Click' events

https://gerrit.wikimedia.org/r/754893

ReleaseTaggerBot edited projects, added MW-1.38-notes (1.38.0-wmf.19; 2022-01-24); removed MW-1.38-notes (1.38.0-wmf.17; 2022-01-10).Jan 18 2022, 1:00 PM
gerritbot added a comment.Jan 18 2022, 1:17 PM

Change 754605 merged by jenkins-bot:

[mediawiki/extensions/GrowthExperiments@wmf/1.38.0-wmf.17] Monitoring: Add '.Save' to distinguish from '.Click' events

https://gerrit.wikimedia.org/r/754605

gerritbot added a comment.Jan 18 2022, 1:17 PM

Change 754906 merged by jenkins-bot:

[mediawiki/extensions/GrowthExperiments@wmf/1.38.0-wmf.18] Monitoring: Add '.Save' to distinguish from '.Click' events

https://gerrit.wikimedia.org/r/754906

Stashbot added a comment.Jan 18 2022, 1:24 PM

Mentioned in SAL (#wikimedia-operations) [2022-01-18T13:24:39Z] <lucaswerkmeister-wmde@deploy1002> Synchronized php-1.38.0-wmf.17/extensions/GrowthExperiments/includes/HomepageHooks.php: Backport: [[gerrit:754605|Monitoring: Add '.Save' to distinguish from '.Click' events (T286366)]] (duration: 00m 54s)

ReleaseTaggerBot edited projects, added MW-1.38-notes (1.38.0-wmf.18; 2022-01-17); removed MW-1.38-notes (1.38.0-wmf.19; 2022-01-24).Jan 18 2022, 2:00 PM
Urbanecm_WMF changed the task status from Open to In Progress.Jan 31 2022, 11:34 AM
kostajh added a comment.Feb 14 2022, 12:52 PM

I'm realizing now from T264460: Enable users to open the first suggested article directly from the mobile task preview that it would be helpful to segment by desktop/mobile, that way we'd get a quicker indicator into how clicks/taps are affected by platform.

kostajh lowered the priority of this task from Medium to Low.Feb 14 2022, 12:52 PM
kostajh moved this task from Code Review to Ready for Development on the Growth-Team (Sprint 0 (Growth Team)) board.
DMburugu added a parent task: T304427: [Epic] Q4 FY 2021-22 Growth Maintenance Work.Mar 29 2022, 4:37 PM
kostajh moved this task from Ready for Development to QA on the Growth-Team (Sprint 0 (Growth Team)) board.Jun 10 2022, 11:38 AM
kostajh added a subscriber: KStoller-WMF.

This can be reviewed at https://grafana.wikimedia.org/d/bSDW0wJ7z/growth-team-product-kpis?orgId=1. The revert rate chart is broken, but I am not going to fix it right now, for reasons I'll elaborate on below.

Overall, I am not sure how useful it is. It's hard to see useful data from the aggregate view on https://grafana.wikimedia.org/d/bSDW0wJ7z/growth-team-product-kpis?orgId=1. And if you filter down to a specific wiki, e.g. https://grafana.wikimedia.org/d/bSDW0wJ7z/growth-team-product-kpis?orgId=1&var-tasktype=All&var-wiki=eswiki&var-structured=All, I am not sure that this provides much useful information either.

@KStoller-WMF @MMiller_WMF perhaps this is something that you can think about, and we could try again once we have a better set of requirements and ideas of things you'd like to see charted out.

Etonkovidova subscribed.Jun 29 2022, 2:54 AM

The issue in T285577: Several wikis have 0 articles for all ORES topics was about fetching articles for topics being broken, so Special:NewcomerTaskinfo displayed all zeroes in Topic table while the Task count was correct.

All topics have exactly zero articles in all task types.

image.png (876×832 px, 67 KB)

The grafana dashboard Growth Team/Growth team product KPIs gives stats on

The data can be viewed (and downloaded) in a tabular form, which is, arguably, is more informative than the graphs. @nettrom_WMF, @KStoller-WMF - anything else needs to be done on this task?

Etonkovidova moved this task from QA to Test in Production | Watching on the Growth-Team (Sprint 0 (Growth Team)) board.Jun 29 2022, 2:55 AM
KStoller-WMF closed this task as Resolved.Jun 29 2022, 6:53 PM

I've been using the Grafana dashboard and I certainly find it very useful to better understand the use of different task types and engagement between wikis. I'm fine with considering this task complete, although someone else should feel free to reopen if they disagree.

Also I'll add a topic to my next 1:1 with @kostajh to chat more about observability, I'm curious to know more about observability and Grafana alerting and if we have some alters set up related to this dashboard.

KStoller-WMF mentioned this in T315898: Growth team product KPIs: Add number of questions sent to mentors to Grafana dashboard.Aug 22 2022, 3:29 PM
kostajh added a comment.Aug 30 2022, 4:23 PM

@KStoller-WMF I updated the dashboard https://grafana.wikimedia.org/d/bSDW0wJ7z/growth-team-product-kpis?orgId=1&var-tasktype=All&var-wiki=All&var-questiontype=All&from=now-7d&to=now&var-variant=All and added sections for:

  • Reverts (by task type)
  • Question posting (T315898)
  • Account registration (though something seems off with the variant recording there, and maybe we want a different visualization there)
KStoller-WMF added a comment.Sep 1 2022, 11:08 PM

This is great, thank you!

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL