In T225110, we recently noticed mediawiki.cirrussearch-request validation errors. It would have been nice to get an alert for these earlier rather than just randomly noticing them.
For eventgate-main events, this will be easier, as we report the HTTP errors when validation fails, as the HTTP response indicates this status. We use 'hasty' mode for eventgate-analytics, so we'll need some other measure of errors.
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | Ottomata | T185233 Modern Event Platform | |||
| Resolved | Ottomata | T201068 Modern Event Platform: Stream Intake Service | |||
| Resolved | Ottomata | T225203 Icinga alert for EventGate validation errors |
Change 514871 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Add monitoring::alerts::kafka_topic_throughput and use it for eventgate validation alerts
Change 514871 merged by Ottomata:
[operations/puppet@production] Add monitoring::alerts::kafka_topic_throughput and use it for eventgate
How to modify EventGate to emit these wasn't obvious, so I changed this task to be for validation errors only. We configure a validation error topic, so we can alert on the throughput of those topics.
Change 516324 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Use method gt instead of ge for eventgate validation error throughput alerts
Change 516324 merged by Ottomata:
[operations/puppet@production] Use method gt instead of ge for eventgate validation error throughput alerts