Common information
- dashboard: https://grafana.wikimedia.org/d/taff979/prometheus-tsdb-cardinality-monitoring?orgId=1&from=now-14d&to=now&timezone=utc&var-prometheus=ext&var-site=eqiad
- description: The rate of new series being created on ext / eqiad has been statistically anomalous for more than 24 hours (z-score: 250.8, threshold: 5). This is a sustained upward deviation from the 3-week baseline and is a strong indicator of label churn, typically caused by high-cardinality label values.
- runbook: https://wikitech.wikimedia.org/wiki/Prometheus#Runbooks
- summary: Anomalous increase in series creation rate on ext (eqiad)
- alertname: PrometheusSeriesCreationRateAnomalyHigh
- prometheus: ext
- recorder: thanos-rule@main
- severity: task
- site: eqiad
- source: thanos
- team: o11y
Firing alerts
- dashboard: https://grafana.wikimedia.org/d/taff979/prometheus-tsdb-cardinality-monitoring?orgId=1&from=now-14d&to=now&timezone=utc&var-prometheus=ext&var-site=eqiad
- description: The rate of new series being created on ext / eqiad has been statistically anomalous for more than 24 hours (z-score: 250.8, threshold: 5). This is a sustained upward deviation from the 3-week baseline and is a strong indicator of label churn, typically caused by high-cardinality label values.
- runbook: https://wikitech.wikimedia.org/wiki/Prometheus#Runbooks
- summary: Anomalous increase in series creation rate on ext (eqiad)
- alertname: PrometheusSeriesCreationRateAnomalyHigh
- prometheus: ext
- recorder: thanos-rule@main
- severity: task
- site: eqiad
- source: thanos
- team: o11y
- Source