Thousands of new metrics are being created in the analytics<hostNum>.*.*.WDQS-Streaming-Updater-POC.* namespace.
At this rate, Graphite will be out of space in about three days. We're out of the woods for now.
Description
Description
Event Timeline
Comment Actions
Mentioned in SAL (#wikimedia-operations) [2020-06-11T03:13:58Z] <shdubsh> removing WDQS-Streaming-Updater-POC metrics on graphite1004 - T255044
Comment Actions
Metrics have been disabled. The urgent issue reported here is fixed, but we still a better way to publish those metrics.
Comment Actions
We enabled metrics again, but removed heavily varying parts of the names, which should drastically reduce the number of newly created ones - once we use start with taskmanager.* and jobmanager.*