Page MenuHomePhabricator

Stop sending per-topic task counts to statsd/Grafana
Closed, ResolvedPublic

Description

In T278411, Growth team added monitoring of the task pool size to a Grafana dashboard. This is used to monitor the healthiness of the team's features.

Currently, the data in statsd is per-topic (meaning it is possible to see how many tasks are available per topic from statsd data). Because that uses a lot of statsd metrics (one per topic, plus one additional one for total), this data is currently only available for link-recommendation.

In T345204: Alert the Growth team when number of available task recommendations drops significantly, we want to add alerting for at least image-recommendation and link-recommendation task counts, which is currently impossible, as image-recommendation data is not available in statsd. In T345204#9128846, I described a backwards-compatible way to add the additional monitoring.

However, I feel like nothing currently used the detailed per-topic data. The only place that I'm aware of is the Grafana dashboard, which has a "Number of topics with <50% pool per wiki (out of 64)" panel, which is currently timeouting (presumably because the number of Add Link-enabled wikis has increased):

image.png (854×1 px, 126 KB)

Since no Growth member complained about this metric being broken, I think we can safely remove it. Proposing to do so in this task.

Event Timeline

Change 953345 had a related patch set uploaded (by Urbanecm; author: Urbanecm):

[mediawiki/extensions/GrowthExperiments@master] linkTaskCounts: Stop producing per-topic statsd data

https://gerrit.wikimedia.org/r/953345

Urbanecm_WMF changed the task status from Open to In Progress.Aug 29 2023, 9:03 PM
Urbanecm_WMF triaged this task as Medium priority.

Change 953345 merged by Urbanecm:

[mediawiki/extensions/GrowthExperiments@master] linkTaskCounts: Stop producing per-topic statsd data

https://gerrit.wikimedia.org/r/953345

Change 957758 had a related patch set uploaded (by Urbanecm; author: Urbanecm):

[mediawiki/extensions/GrowthExperiments@wmf/1.41.0-wmf.26] linkTaskCounts: Stop producing per-topic statsd data

https://gerrit.wikimedia.org/r/957758

Change 957758 merged by Urbanecm:

[mediawiki/extensions/GrowthExperiments@wmf/1.41.0-wmf.26] linkTaskCounts: Stop producing per-topic statsd data

https://gerrit.wikimedia.org/r/957758

Mentioned in SAL (#wikimedia-operations) [2023-09-14T15:47:52Z] <urbanecm@deploy1002> Started scap: Backport for [[gerrit:957396|listTaskCounts: Push total task counts to statsd for all tasks (T345204)]], [[gerrit:957758|linkTaskCounts: Stop producing per-topic statsd data (T345210)]]

Mentioned in SAL (#wikimedia-operations) [2023-09-14T15:55:29Z] <urbanecm@deploy1002> Finished scap: Backport for [[gerrit:957396|listTaskCounts: Push total task counts to statsd for all tasks (T345204)]], [[gerrit:957758|linkTaskCounts: Stop producing per-topic statsd data (T345210)]] (duration: 07m 37s)