At some point in the past, the GC metrics collected switched from Gauge to Histogram, and the metric names changed from something like: nodejs_gc_incremental_duration_seconds_bucket to nodejs_gc_duration_seconds_bucket{kind="incremental"}. Now that service-utils have these metrics enabled by default, we should fix the dashboard.
Description
Description
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | tchin | T360924 Replace service runner with a simplified library to better support metrics and debugging: service-utils | |||
| Resolved | tchin | T361769 Migrate and re-deploy eventstreams using service-utils | |||
| Resolved | tchin | T386204 Update eventstreams Grafana Dashboards to use histogram for router metrics | |||
| Declined | gmodena | T350180 Upgrade prom-client in NodeJS service-runner and enable collectDefaultMetrics | |||
| Resolved | tchin | T386709 Update eventstreams Grafana Dashboards to fix GC metrics |
Event Timeline
Comment Actions
There doesn't seem to be a service label on these GC metrics so I used kubernetes_namespace instead.
GC Rate Average
- avg (rate(nodejs_gc_incremental_duration_seconds_count{job="k8s-pods", service="${service}"}[5m])) - avg (rate(nodejs_gc_major_duration_seconds_count{job="k8s-pods", service="${service}"}[5m])) - avg (rate(nodejs_gc_minor_duration_seconds_count{job="k8s-pods", service="${service}"}[5m])) + avg (rate(nodejs_gc_duration_seconds_count{kind="incremental", kubernetes_namespace="$service"}[5m])) + avg (rate(nodejs_gc_duration_seconds_count{kind="major", kubernetes_namespace="$service"}[5m])) + avg (rate(nodejs_gc_duration_seconds_count{kind="minor", kubernetes_namespace="$service"}[5m]))
GC Rate Max
- max (rate(nodejs_gc_incremental_duration_seconds_count{job="k8s-pods", service="${service}"}[5m])) - max (rate(nodejs_gc_major_duration_seconds_count{job="k8s-pods", service="${service}"}[5m])) - max (rate(nodejs_gc_minor_duration_seconds_count{job="k8s-pods", service="${service}"}[5m])) + max (rate(nodejs_gc_duration_seconds_count{kind="incremental", kubernetes_namespace="$service"}[5m])) + max (rate(nodejs_gc_duration_seconds_count{kind="major", kubernetes_namespace="$service"}[5m])) + max (rate(nodejs_gc_duration_seconds_count{kind="minor", kubernetes_namespace="$service"}[5m]))
GC Duration Average
- sum (rate(nodejs_gc_incremental_duration_seconds_sum{job="k8s-pods", service="$service"}[5m]))/sum (rate(nodejs_gc_incremental_duration_seconds_count{job="k8s-pods", service="$service"}[5m])) - sum (rate(nodejs_gc_major_duration_seconds_sum{job="k8s-pods", service="$service"}[5m]))/sum (rate(nodejs_gc_major_duration_seconds_count{job="k8s-pods", service="$service"}[5m])) - sum (rate(nodejs_gc_minor_duration_seconds_sum{job="k8s-pods", service="$service"}[5m]))/sum (rate(nodejs_gc_minor_duration_seconds_count{job="k8s-pods", service="$service"}[5m])) + sum (rate(nodejs_gc_duration_seconds_sum{kind="incremental", kubernetes_namespace="$service"}[5m]))/sum (rate(nodejs_gc_duration_seconds_count{kind="incremental", kubernetes_namespace="$service"}[5m])) + sum (rate(nodejs_gc_duration_seconds_sum{kind="major", kubernetes_namespace="$service"}[5m]))/sum (rate(nodejs_gc_duration_seconds_count{kind="major", kubernetes_namespace="$service"}[5m])) + sum (rate(nodejs_gc_duration_seconds_sum{kind="minor", kubernetes_namespace="$service"}[5m]))/sum (rate(nodejs_gc_duration_seconds_count{kind="minor", kubernetes_namespace="$service"}[5m]))
GC Duration Quantiles
- histogram_quantile(0.5, sum(rate(nodejs_gc_incremental_duration_seconds_bucket{job="k8s-pods", service="$service"}[5m])) by (le)) - histogram_quantile(0.9, sum(rate(nodejs_gc_incremental_duration_seconds_bucket{job="k8s-pods", service="eventstreams"}[5m])) by (le)) - histogram_quantile(0.99, sum(rate(nodejs_gc_incremental_duration_seconds_bucket{job="k8s-pods", service="eventstreams"}[5m])) by (le)) - histogram_quantile(0.5, sum(rate(nodejs_gc_major_duration_seconds_bucket{job="k8s-pods", service="eventstreams"}[5m])) by (le)) - histogram_quantile(0.9, sum(rate(nodejs_gc_major_duration_seconds_bucket{job="k8s-pods", service="eventstreams"}[5m])) by (le)) - histogram_quantile(0.99, sum(rate(nodejs_gc_major_duration_seconds_bucket{job="k8s-pods", service="eventstreams"}[5m])) by (le)) - histogram_quantile(0.5, sum(rate(nodejs_gc_minor_duration_seconds_bucket{job="k8s-pods", service="eventstreams"}[5m])) by (le)) - histogram_quantile(0.9, sum(rate(nodejs_gc_minor_duration_seconds_bucket{job="k8s-pods", service="eventstreams"}[5m])) by (le)) - histogram_quantile(0.99, avg(rate(nodejs_gc_minor_duration_seconds_bucket{job="k8s-pods", service="eventstreams"}[5m])) by (le)) + histogram_quantile(0.5, sum(rate(nodejs_gc_duration_seconds_bucket{kind="incremental", kubernetes_namespace="$service"}[5m])) by (le)) + histogram_quantile(0.9, sum(rate(nodejs_gc_duration_seconds_bucket{kind="incremental", kubernetes_namespace="$service"}[5m])) by (le)) + histogram_quantile(0.99, sum(rate(nodejs_gc_duration_seconds_bucket{kind="incremental", kubernetes_namespace="$service"}[5m])) by (le)) + histogram_quantile(0.5, sum(rate(nodejs_gc_duration_seconds_bucket{kind="major", kubernetes_namespace="$service"}[5m])) by (le)) + histogram_quantile(0.9, sum(rate(nodejs_gc_duration_seconds_bucket{kind="major", kubernetes_namespace="$service"}[5m])) by (le)) + histogram_quantile(0.99, sum(rate(nodejs_gc_duration_seconds_bucket{kind="major", kubernetes_namespace="$service"}[5m])) by (le)) + histogram_quantile(0.5, sum(rate(nodejs_gc_duration_seconds_bucket{kind="minor", kubernetes_namespace="$service"}[5m])) by (le)) + histogram_quantile(0.9, sum(rate(nodejs_gc_duration_seconds_bucket{kind="minor", kubernetes_namespace="$service"}[5m])) by (le)) + histogram_quantile(0.99, sum(rate(nodejs_gc_duration_seconds_bucket{kind="minor", kubernetes_namespace="$service"}[5m])) by (le))
- For some reason the last histogram_quantile had an avg instead of a sum, I assume that was a type and I fixed it.
GC Incremental Duration Average - Histogram
- avg (rate(nodejs_gc_incremental_duration_seconds_bucket{job="k8s-pods", service="${service}"}[5m])) + avg (rate(nodejs_gc_duration_seconds_bucket{kind="incremental", kubernetes_namespace="$service"}[5m]))
GC Major Duration Average - Histogram
- avg (rate(nodejs_gc_major_duration_seconds_bucket{job="k8s-pods", service="${service}"}[5m])) + avg (rate(nodejs_gc_duration_seconds_bucket{kind="major", kubernetes_namespace="$service"}[5m]))
GC Minor Duration Average - Histogram
- avg (rate(nodejs_gc_minor_duration_seconds_bucket{job="k8s-pods", service="${service}"}[5m])) + avg (rate(nodejs_gc_duration_seconds_bucket{kind="minor", kubernetes_namespace="$service"}[5m]))
GC Incremental Duration Max - Histogram
- max (rate(nodejs_gc_incremental_duration_seconds_bucket{job="k8s-pods", service="${service}"}[5m])) + max (rate(nodejs_gc_duration_seconds_bucket{kind="incremental", kubernetes_namespace="$service"}[5m]))
GC Major Duration Max - Histogram
- max (rate(nodejs_gc_major_duration_seconds_bucket{job="k8s-pods", service="${service}"}[5m])) + max (rate(nodejs_gc_duration_seconds_bucket{kind="major", kubernetes_namespace="$service"}[5m]))
GC Minor Duration Max - Histogram
- max (rate(nodejs_gc_minor_duration_seconds_bucket{job="k8s-pods", service="${service}"}[5m])) + max (rate(nodejs_gc_duration_seconds_bucket{kind="minor", kubernetes_namespace="$service"}[5m]))