Update eventstreams Grafana Dashboards to use histogram for router metrics
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	tchin
	Feb 12 2025, 2:27 PM

Project Tags

Referenced Files

None

Subscribers

Description

eventstreams (and eventstreams-internal) used a version of service-template-node that is so old that the router metrics are in a Gauge format. The current version of service-template-node and service-utils uses Histogram instead. All express router metrics in Grafana must be updated.
Here is what I think is the safest way to go about this:

1. Deploy eventstreams to codfw
2. Wait and see which Grafana charts go to 0
3. Update those charts, see if the data comes back
3a. Track what I changed on this ticket just in case we have to revert
4. Deploy eventstreams to eqiad

Related Objects
Search...

Status	Assigned	Task
Resolved	tchin	T360924 Replace service runner with a simplified library to better support metrics and debugging: service-utils
Resolved	tchin	T361769 Migrate and re-deploy eventstreams using service-utils
Resolved	tchin	T386204 Update eventstreams Grafana Dashboards to use histogram for router metrics
Resolved	tchin	T386709 Update eventstreams Grafana Dashboards to fix GC metrics

Event Timeline

tchin created this task.Feb 12 2025, 2:27 PM

Restricted Application removed a project: Patch-For-Review. · View Herald TranscriptFeb 12 2025, 2:27 PM

Ottomata awarded a token.Feb 12 2025, 2:33 PM

tchin changed the task status from Open to In Progress.Feb 13 2025, 6:20 PM

tchin updated the task description. (Show Details)Feb 13 2025, 6:30 PM

Stream Connection Duration

- min(express_router_request_duration_seconds{service="$service", path=~"v2/stream/.*"})
+ min(rate(express_router_request_duration_seconds_sum{service="$service", path=~"stream/.*"}[5m])/rate(express_router_request_duration_seconds_count{service="$service", path=~"stream/.*"}[5m])>0)

Traffic

by HTTP method
- count by (method) (rate(express_router_request_duration_seconds{service="$service"}[5m]))
+ sum by (method) (rate(express_router_request_duration_seconds_count{service="$service",status=~"\\d\\d\\d"}[5m]))

by endpoint
- count by (path) (rate(express_router_request_duration_seconds{service="$service"}[5m]))
+ sum by (path) (rate(express_router_request_duration_seconds_count{service="$service",status=~"\\d\\d\\d"}[5m]))

by HTTP status
- count by (status) (rate(express_router_request_duration_seconds{service="$service"}[5m]))
+ sum by (status) (rate(express_router_request_duration_seconds_count{service="$service"}[5m]))

Errors

HTTP errors
- sum by (status) (rate(express_router_request_duration_seconds{service="$service", status=~"4..|5.."}[5m]))
+ sum by (status) (rate(express_router_request_duration_seconds_count{service="$service", status=~"4..|5.."}[5m]))

Latency

I don't know why, but it seems like changing these went from minutes on the y axis to µs. These expressions are the same for EventGate though so I assume they're correct

Total
- avg (rate(express_router_request_duration_seconds{service="$service"} [5m]))
+ avg (rate(express_router_request_duration_seconds_sum{service="$service",status=~"\\d\\d\\d"}[5m])/rate(express_router_request_duration_seconds_count{service="$service",status=~"\\d\\d\\d"}[5m])>0)

by HTTP method
- avg by (method) (rate(express_router_request_duration_seconds{service="$service"} [5m]))
+ avg by (method) (rate(express_router_request_duration_seconds_sum{service="$service",status=~"\\d\\d\\d"}[5m])/rate(express_router_request_duration_seconds_count{service="$service",status=~"\\d\\d\\d"}[5m])>0)

by endpoint
- avg by (path) (rate(express_router_request_duration_seconds{service="$service"} [5m]))
+ avg by (path) (rate(express_router_request_duration_seconds_sum{service="$service",status=~"\\d\\d\\d"}[5m])/rate(express_router_request_duration_seconds_count{service="$service",status=~"\\d\\d\\d"}[5m])>0)

by HTTP status
- avg by (status) (rate(express_router_request_duration_seconds{service="$service"} [5m]))
+ avg by (status) (rate(express_router_request_duration_seconds_sum{service="$service"}[5m])/rate(express_router_request_duration_seconds_count{service="$service"}[5m])>0)

Saturation

Max Memory
- max (nodejs_process_heap_used_bytes{service="$service"})
+ max (nodejs_process_heap_used_bytes)

- max (nodejs_process_heap_total_bytes{service="$service"})
+ max (nodejs_process_heap_total_bytes)

- max (nodejs_process_heap_rss_bytes{service="$service"})
+ max (nodejs_process_heap_rss_bytes)

There's a few charts that seem to have been broken since who knows how long ago, like Top 5 pod Memory (current) and all the GC * stuff that I left alone. The current error rate has also been 0 since the time I've been staring at the dashboards, so unsure if that's been broken or if we just have a very good service

tchin moved this task from Next Up to Ready to Deploy on the Data-Engineering (Q3 2025 January 1st - March 31th) board.Feb 13 2025, 7:02 PM

The GC stuff could be fixed by T350180: Upgrade prom-client in NodeJS service-runner and enable collectDefaultMetrics. Could we do that in service-utils?

Garbage collection is enabled by default, but it seems that the default metrics are also in histogram now and not gauge which is what the dashboard assumes

Great! I closed that other task then. Thanks!

Ahoelzl moved this task from Ready to Deploy to Done on the Data-Engineering (Q3 2025 January 1st - March 31th) board.Feb 20 2025, 6:22 PM

Ahoelzl closed subtask T386709: Update eventstreams Grafana Dashboards to fix GC metrics as Resolved.Apr 3 2025, 3:22 PM

tchin added a project: service-utils.Jul 2 2026, 4:17 AM