Page MenuHomePhabricator

Paypal vault API rate limiting through Gravy 2025-08-25 and 09-04
Closed, ResolvedPublic

Description

This happened on 2025-08-25 which was day 1 en6c email send and second week of sending volume through gravy paypal

@AMJohnson noticed that we were getting failures in the gravy console with no error messages and when digging into them further they had an error on the vault api with 429
service_rate_limit

example gravy id 0829fa34-a7f4-46a7-aaba-625776700ebf

Gravy got in contact with paypal and upped our rate limit but the traffic had trailed off too in the meantime

Response from paypal
"When looking at our logs for your Client ID, I am seeing a spike in the 429 errors between 0900-1000 Pacific today. The errors have stopped after that, and we are now seeing normal processing results on your Client ID. If you recently went live, it may have inadvertently triggered a rate limiter on our end, but since our rate limiter dynamically flexes with traffic, it may have auto-corrected."

This happened again on 2025-09-04, two donors emailed about it. I have sent PayPal an email asking if our limit can be increased as we don't have consistent traffic

Event Timeline

Cstone renamed this task from Paypal vault API rate limiting through Gravy to Paypal vault API rate limiting through Gravy 2025-08-25.Aug 28 2025, 3:44 PM

Potential fallout from this issue. Pasting note from earlier call:

cstone, before we got distracted with Gravy JS, there was a question/worry about the PayPal rate-limited transactions. In short, it appears tha Gravy has marked some of the affected items as captured making me wonder if they actually got captured or not. However, the ones marked as captured were not included in the Gravy audit, and they didn't get in via the usual process due to the errors, meaning it looks like we might have another batch of snagged transactions that needs manually importing.

Related affected transaction is here https://wikimedia.gr4vy.app/merchants/default/transactions/3845a41f-55fd-4edc-9331-0d44f3ac2b6f/overview and was looked at as part of discussions on T402841: recent Gravy / PayPal donations missing PII in Civi

Cstone renamed this task from Paypal vault API rate limiting through Gravy 2025-08-25 to Paypal vault API rate limiting through Gravy 2025-08-25 and 09-04.Sep 4 2025, 7:35 PM
Cstone updated the task description. (Show Details)

This happened again today, ~210 transactions were stopped by it in a 36 minute time window:

First one September 4th, 2025 4:02 PM
https://wikimedia.gr4vy.app/merchants/default/transactions/1046c6e4-4b9e-43a1-90f1-7356fe8eee49/overview

Last one
September 4th, 2025 4:38 PM UTC
https://wikimedia.gr4vy.app/merchants/default/transactions/08502e10-cc6f-402d-9229-0f7b7e561b1d/overview

After lots of email threads, Paypal said our account has been updated to have an exception to the rate limit

XenoRyet set Final Story Points to 4.