As of this AM we're collecting syslog total message count, and dropped message count, to prometheus. They're on the main fundraising dashboard here: https://grafana.wikimedia.org/dashboard/db/fundraising-overview?refresh=1m&orgId=1. We haven't figured out alerting from prometheus/grafana yet, but that's the logical next step.

In T176924#3651256, @Jgreen wrote:

As of this AM we're collecting syslog total message count, and dropped message count, to prometheus. They're on the main fundraising dashboard here: https://grafana.wikimedia.org/dashboard/db/fundraising-overview?refresh=1m&orgId=1. We haven't figured out alerting from prometheus/grafana yet, but that's the logical next step.

Also at this stage it would be impractical to alert because the alarm would be going off constantly for the civicrm host, where queue consumers are still pushing too much log traffic.

Darn, I'd hoped the Civi host would be fine after cutting the message rate in half, but it looks like there are still spikes of rate-limiting