Page MenuHomePhabricator

Add prometheus metrics collection for Istio and Knative
Open, Needs TriagePublic

Description

Istio and knative offer Prometheus metrics out of the box, we should verify it and start collecting metrics from them.

Ideally at the end of this task we should have two nice dashboards with basic metrics.

Event Timeline

As Joe pointed out in T287007#7224824, we are indeed already collecting Istio metrics!

Added basic metrics to the istio dashboard (rps, bytes, latency) broken down by backend. The dashboard is still very raw but it is returning good info, let's proceed with knative before making it perfect (that will come with time).

Knative doesn't seem to offer prometheus annotations, but the number of metrics for serving is really nice: https://knative.dev/docs/admin/collecting-metrics/serving-metrics/metrics/ (this is the last version of course, 0.18 may have less). There is also a config-map for observability, listing a lot of nice options like having access logs etc..

Change 731875 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/deployment-charts@master] knative-serving: add prometheus annotations

https://gerrit.wikimedia.org/r/731875

Change 731875 merged by Elukey:

[operations/deployment-charts@master] knative-serving: add prometheus annotations

https://gerrit.wikimedia.org/r/731875