Page MenuHomePhabricator

Gravy capture jobs getting lost... maybe
Closed, ResolvedPublic

Description

We've had a few gravy donations flagged by donor relations, which were captured and settled at the backend processor, but that did not make it into CiviCRM. There have been a few explanations: Chinese characters breaking API calls and initial fraud scores rejecting transactions that Adyen later authorised.

However, some transactions remain unresolved and have required manual re-importing. When we trace the logs for these cases, we see that a transaction starts with a capture_pending status. The pending transaction resolver picks it up and later drops it because its status has already moved to capture_succeeded, suggesting it doesn't need resolving. The problem is that our system never recorded that successful capture, even though we received the capture_succeeded webhook from Gravy. This suggests the capture job itself is either not being created or is failing to process correctly.

A recent example is https://wikimedia.gr4vy.app/merchants/default/transactions/8b79cf90-1a4d-4872-872c-ba1ac3ef9e39/overview. If you follow the history of this transaction before Sept 4th, you will see it gets lost. I had to manually import it into CiviCRM on September 4th because I couldn't understand why it hadn't been imported initially. Therefore, log data from after that date won't be helpful.

Happy hunting!

Event Timeline

We've added some additional logging to the pending transaction resolver here to help identify more easily in the logs the donation that the resolver drops early due to deciding it's already been resolved.

Cstone reassigned this task from Cstone to AnnWF.
Cstone added a subscriber: AnnWF.
Cstone subscribed.

Both transactions below are not appearing in Civi after manual settlement:

*CID 65047211, Gravy transaction 73d1fd1c-8962-4e12-b90c-f4053bdfab63 → settled on Sept 4

*CID 67686815, Gravy transaction 7cf852f6-fc86-428c-bbd6-799833ce90ed → also settled on Sept 4

Additionally, a second transaction from CID 67686815 (d1605d0a-5fbc-47c8-a90b-ffd4edbf602a) is stuck in processing at Gravy, while Adyen shows it as canceled.

@SHust, The timestamps of these transactions made me think they might have been affected by a recent Gravy outage, and it turns out they were.

Unfortunately, we can't use the usual importer tool to bring these into CiviCRM, as we don't have the necessary log data available to us to rebuild the donation messages. The Gravy outage resulted in incomplete and partial API responses, so we're currently working with Gravy to figure out the best way to pull these in.

Both transactions below are not appearing in Civi after manual settlement:

*CID 65047211, Gravy transaction 73d1fd1c-8962-4e12-b90c-f4053bdfab63 → settled on Sept 4

*CID 67686815, Gravy transaction 7cf852f6-fc86-428c-bbd6-799833ce90ed → also settled on Sept 4

Additionally, a second transaction from CID 67686815 (d1605d0a-5fbc-47c8-a90b-ffd4edbf602a) is stuck in processing at Gravy, while Adyen shows it as canceled.

Another one guessing it's probably the same thing @jgleeson. Donor in #1723321 I manually settled their transaction b7b12b49-dc84-4108-9c42-a701668b6a24 on 09/04 and it's not in Civi.

AnnWF removed AnnWF as the assignee of this task.Sep 16 2025, 5:27 PM
AnnWF moved this task from In Progress to Backlog on the Fundraising Tech - Chaos Crew board.

Hi @jgleeson, just checking to see if there are any updates here. The CIDs 65047211 and 67686815 still don’t show their recent donations in Civi.

Hi @SHust. Bad news, unfortunately. Although I can see that those two transactions are listed on the Gravy console as settled, we haven't received a notification of the settlement/capture event. I'll chase it up on Slack and explain the issue.

I manually added b7b12b49-dc84-4108-9c42-a701668b6a24 and sent a TY email to ZD #1723321 / cid=23581948

XenoRyet triaged this task as Medium priority.Oct 14 2025, 7:59 PM

Did we get the webhook notifications in these cases?

Not sure if related but here are two I saw with bad gravy responses T405692: Transactions that didn't make it to civi due to gravy error

XenoRyet raised the priority of this task from Medium to High.Oct 22 2025, 4:55 PM

Thanks @Cstone. That might have caused the same issue. I think we're good to close this one. It looks like there are scenarios where stuff breaks, and we miss an import, but it doesn't look like there's a silver bullet to fix them, so I'm thinking we can close this and keep an eye on new instances.

XenoRyet set Final Story Points to 4.