Page MenuHomePhabricator

Many new metrics in Graphite for WDQS-Streaming-Updater-POC
Closed, ResolvedPublic

Description

Thousands of new metrics are being created in the analytics<hostNum>.*.*.WDQS-Streaming-Updater-POC.* namespace.

At this rate, Graphite will be out of space in about three days. We're out of the woods for now.

Event Timeline

From discussion in -analytics, @dcausse indicated that they are safe to remove.

Mentioned in SAL (#wikimedia-operations) [2020-06-11T03:13:58Z] <shdubsh> removing WDQS-Streaming-Updater-POC metrics on graphite1004 - T255044

Metrics have been disabled. The urgent issue reported here is fixed, but we still a better way to publish those metrics.

We enabled metrics again, but removed heavily varying parts of the names, which should drastically reduce the number of newly created ones - once we use start with taskmanager.* and jobmanager.*