We recently released the Gravy Error Tracker as part of the work on T387411: How will we expose payment processor errors/issues with gravy, which provides us with real-time email alerts of payment error spikes encountered during Gravy transactions.

Donor Relations is interested in understanding the possibility of providing real-time processing anomaly detection across the wider stack to better inform their advice to donors during ongoing issues. This enhanced monitoring would need to be processor-agnostic and track anomalies across the full stack, including Payments and CiviCRM.

@krobinson has provided specific examples of the issues where improved alerting would have helped: (https://docs.google.com/document/d/1sZTvQxGnqaRahpP_P2i4iORwfhj9qM7axv-Tl_j-ifw/edit?tab=t.0).

Some ideas & thoughts:

• Real-Time Alerting: We could adapt our existing tooling so fr-tech systems can report detected anomalies as close to real-time as possible. These alerts could be channelled to key stakeholder teams via automated email or Slack messages.
• Monitoring and Visualisation: We could begin outputting statistics on these events to be consumed by Prometheus. This would allow us to visualise the anomaly data in Grafana for technical teams.
• Enhanced Reporting: There is also interest in consuming this data in Metabase for broader reporting. To enable this and improve overall context, we could investigate enhancing the amount of error detail saved in CiviCRM whenever a processing dropout or anomaly is encountered.