Page MenuHomePhabricator

Event Logging data is not showing up in Graphite anymore since last week
Closed, ResolvedPublic

Description

http://graphite.wikimedia.org/ is not showing any data for any event logging metrics since April 29th.

We looked at our reporter and that looked to be sending stuff to statsd, just wondering if you could help us debug.

Event Timeline

Milimetric raised the priority of this task from to Needs Triage.
Milimetric updated the task description. (Show Details)
Milimetric added a subscriber: Milimetric.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 6 2015, 5:51 PM
chasemp added a subscriber: chasemp.May 6 2015, 6:01 PM

I see this in icinga for graphite1001:

Throughput of event logging events CRITICAL: 92.86% of data above the critical threshold [600.0]

thanks for the report @Milimetric! indeed the culprit was due to multiple interactions/renaming with: timers, simple counters and extended counters and the fact that a single metric.wsp will shadow and metric/ directory at the same level in graphite.

I've set aside the deprecated simple counters under another name and thus the full extended counters now show up:

http://graphite.wikimedia.org/render/?width=586&height=308&_salt=1431007652.126&target=eventlogging.overall.raw.rate

fgiunchedi triaged this task as Normal priority.
fgiunchedi set Security to None.
Milimetric closed this task as Resolved.May 7 2015, 2:40 PM

thanks very much @fgiunchedi, I'm not sure what that means but I'll try to get Andrew to translate :)

@Milimetric no problem -- I'll give a bit more context :) we've switched statsd implementation from txstatsd to statsite for performance/efficiency reasons, this meant renaming some metrics to match what statsite emits to graphite and deprecate metric types not supported by statsite. tl;dr the txstatsd -> statsite transition, coupled with the fact that graphite won't show metric directories if a metric file already exists with the same name, was masking the updated metrics from showing up

kevinator moved this task from Next Up to Done on the Analytics-Kanban board.May 19 2015, 7:35 PM