We already emit a validation error event to Kafka, but it'd be nice to emit some metrics (via prometheus) about validation errors per stream and schema uri. To do this, we need to update EventGate to use service-runner with Prometheus support, which I believe has not yet been released.
We should be able to do this by passing in the service-runner metrics object to eventgate-wikimedia's makeMapToErrorEvent function and emitting the metric then.
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | Ottomata | T272714 EventGate should use recent service-runner (^2.8.1) with Prometheus support | |||
| Resolved | Ottomata | T257237 eventgate-wikimedia should emit metrics about validation errors |
Hm, we might be able to do this incrementally. service-runner will let us configure multiple metrics clients. We can keep the existing statsd -> prometheues-statsd-exporter stuff, and at the same time use service-runner's prometheus for this validation error metric. This way, we wouldn't have to re-implement node-rdkafka-statsd as node-rdkafka-prometheus. This would allow us to incrementally move away from prometheus-statsd-exporter.
However, we'd have to run service-runner's prometheus http server on a different port than the one that prometheus-statsd-exporter uses, and I'm not sure our k8s configuration would be happy having to scrape 2 different prometheus endpoints for one service.
I guess that's a Q for service-ops folks. @JMeybohm, @akosiaris:
Would it be difficult to scrape 2 different prometheus endpoints for one service in k8s? One for prometheus-statsd-exporter, and another for service-runner prometheus?
Or, I could probably just get this working with the existent service-runner statsd + prometheus-statsd-exporter config, and just define a mapping from statsd -> prometheus like we do in the helm chart now.
Change 657902 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[eventgate-wikimedia@master] Emit error.validation counter metrics per stream
Change 657908 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/deployment-charts@master] [WIP] eventgate - Map from eventgate event and error statsd metrics to prometheus
Change 657902 merged by Ottomata:
[eventgate-wikimedia@master] Emit metrics about events and validation errors
Change 658410 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/deployment-charts@master] eventgate-analytics-external - bump to 2021-01-25-183848-production
Change 658410 merged by Ottomata:
[operations/deployment-charts@master] eventgate-analytics-external - bump to 2021-01-25-183848-production
Change 657908 merged by Ottomata:
[operations/deployment-charts@master] eventgate - Map from eventgate event and error statsd metrics to prometheus
Change 658412 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/deployment-charts@master] eventgate-* - bump to 2021-01-25-183848-production
Change 658412 merged by Ottomata:
[operations/deployment-charts@master] eventgate-* - bump to 2021-01-25-183848-production
@fgiunchedi hello! Been reading some alert documentation stuff and I have some questions.
I want to add an alert to this panel that will include information about validation error rate per stream going over some threshold. The last time I added an alert it was done via monitoring::check_prometheus in Puppet. I could probably get this to work there, but I'd like the alert to be a bit more dynamic, reporting hopefully WHICH steams have validation errors, not just that there are some.
I'll try to find you on IRC sometime this week for help! :)
While possible, it's a bit tricky to do. Prometheus service discovery would need to be configured to look at a second (third, actually as there is a metrics endpoint for envoy as well) set of annotations (https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/refs/heads/production/modules/profile/manifests/prometheus/k8s.pp#221).
As this seems like a temporary solution, I would prefer a way around that.
For sure! I'll add a bit of context below as well:
At the moment there are several options for "alert from Prometheus metrics" use case, with various degrees of legacyness (sp?) and ease of use, namely:
In terms of workflow, using AM means that alerts will show up at https://alerts.wikimedia.org and must be ack'd from there. Notifications can happen via IRC and/or email, usually depending on the alert' severity.
Hope that helps! Let's sync up on IRC too.
Change 661999 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Add alerts on eventgate_validation_errors_total rate for each eventgate service
Change 661999 merged by Ottomata:
[operations/puppet@production] Add alerts on eventgate_validation_errors_total rate for each eventgate service