Page MenuHomePhabricator
Create Task
Maniphest T97640

MediaWiki jobs statsd metric are no more monitored due to a metric name change
Closed, DuplicatePublic
Actions

Description

The Gdash jobq page shows empty graph. The images refers to metrics such as MediaWiki.stats.job-insert.count.

In puppet, we have some graphite monitoring at modules/mediawiki/manifests/monitoring/graphite.pp for:

MediaWiki.stats.job-insert.count
MediaWiki.stats.job-pop.count

Looking on https://graphite.wikimedia.org/ the MediaWiki.stats. hierarchy has disappeared. End result: gdash are empty and we have loose all jobq monitoring.

The stats are still available and inserted but directly under MediaWiki (ie .stats has been dropped somehow).

Looking at MediaWiki.job-insert.count
From 29/04/2015 09:00 Until 29/04/2015 12:00:00

statsd-job-insert.png (308×866 px, 35 KB)

The change occurred on 29/04/2015 at 9:47am UTC.

Related Objects

Event Timeline

hashar created this task.Apr 30 2015, 12:58 PM
hashar raised the priority of this task from to Needs Triage.
hashar updated the task description. (Show Details)
hashar added projects: acl*sre-team, MediaWiki-Logevents, Grafana.
hashar subscribed.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 30 2015, 12:58 PM
hashar updated the task description. (Show Details)Apr 30 2015, 1:03 PM
hashar set Security to None.
Krenair subscribed.Apr 30 2015, 1:06 PM
hashar added subscribers: fgiunchedi, ori.Apr 30 2015, 1:20 PM

The Statsd has been changed recently https://gerrit.wikimedia.org/r/#/c/191854/ by @ori but that seems to have been deployed a while ago.

During that morning @fgiunchedi send a couple statsd related commit in operations/puppet:

Maybe that caused the metric names change?

fgiunchedi added a comment.Apr 30 2015, 2:11 PM

I think there's two factors at play, https://gerrit.wikimedia.org/r/#/c/206781/ dealt with the extended counters which indeed renamed the metrics, however the stats prefix has been long gone from mediawiki (can't find the commit ATM) so previously it would have been MediaWiki.job-insert.
related is also T85641 and T87594

fgiunchedi closed this task as a duplicate of T87594: Job queue stats are broken.Apr 30 2015, 2:28 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits