From @jgleeson "I see ~200 transactions affected with the same response during the 18-second window"
from gravy " we’ve identified that the errors you observed coincided with a short-lived database issue around 9:09:47 AM CST. The DB auto-recovered during that window."
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Open | None | T403769 Gravy Chaos Epic | |||
| Resolved | jgleeson | T403773 Gravy database outage 2025-09-04 |
I pulled the affected transactions out of the logs and managed to cobble together a list of Gravy URLs here by appending the ct_id to the end of a Gravy transaction search URL. After reviewing a few records, I can see that not all of them have transaction pages. I'm guessing not all payment methods behave the same way when the transaction fails, but it will at least give us an easier way to refer to examples when discussing the issue with Gravy.
I tried finding some of the affected records in our audit from last night, and although a few are in there, they are all stuck on authorization_succeeded, which means they won't have been pulled into CiviCRM and will still need to be manually captured. I've asked Gravy to provide us with a list on Slack here and then we can talk with Donor Services about the best way to capture & import them.
From DR's perspective capturing authorized transactions after this much time doesn't scale that well (donors may have used alt methods, may not be expecting a charge to affect their balances) so we'll concentrate on importing only the ones from this batch that are settled.
Of the 58 transactions marked as Gravy-side fail, only 9 settled at Gravy, and they all reached Civi on the 4th.