Page MenuHomePhabricator

Donor donated 3 times in 7 seconds with gravy google pay and had a duplicate invoice id
Closed, ResolvedPublic

Description

Donor cid=70080516 made three donations via Adyen/Gravy on March 6th, 2026, but only two reached Civi. It's a little odd in that two of the three have the same external identifier, 245260402.2.

Piero confirmed that Gravy sent a webhook for each of them, so can we determine why the third didn't reach Civi?

https://wikimedia.gr4vy.app/merchants/default/transactions/c393558a-7a08-4cfa-894b-ed99dadb1423/overview is the one that didn't get picked up.

I'm going to refund two of them for the donor soon, but if that would impede the investigation please let me know.

2026-03-18:
Possible other instance of this
cid=69250764

More examples from older task:
GRAVY e0e34b64-14be-487c-bdff-db63b8f8f0ba
GRAVY ea3b86ed-2786-4e6e-b982-60eec15b1f59

December 6th, 2025 5:54 AM
December 6th, 2025 5:55 AM

Another one is
cid=69250764
cid=52667387 not sure what's going on with this one, the amounts are unique that the donor had to type them

Another one, a recurring from 2026-01-12
cid=70047497
cb93cff2-211a-40eb-87c9-3984fa0a6f9c
dd7bb9ee-7b49-434a-acce-6ecc2051593f

Event Timeline

@MBeat33 you can refund them, that won't affect the investigation

Two issues here:

1 - Donor was able to donate 3 times in 7 seconds from the front end

2 - The audit ingestion didn't grab the third transaction (I'm guessing due to the duplicate id you saw @MBeat33 )

I'm going to rename this task to the bug itself

Cstone renamed this task from stray Gravy donation from March 6th not in Civi to Donor donated 3 times in 7 seconds with gravy google pay and had a duplicate invoice id.Mar 11 2026, 5:04 AM
Cstone moved this task from Backlog to In Progress on the Fundraising Tech - Chaos Crew board.
Eileenmcnaughton renamed this task from Donor donated 3 times in 7 seconds with gravy google pay and had a duplicate invoice id to Donor donated 3 times in 7 seconds with gr4vy google pay and had a duplicate invoice id.Mar 16 2026, 7:02 AM

We made 3 separate full calls to gravy:

payments-gravy-20260307.gz:Mar 6 11:09:20 payments1005 gravy_gateway: 245260402:245260402.2 Calling approvePayment on PSP reference 1ee7a916-ce2a-4522-8c7a-f706a5a30be0
payments-gravy-20260307.gz:Mar 6 11:09:22 payments1005 gravy_gateway: 245260402:245260402.2 Calling approvePayment on PSP reference c393558a-7a08-4cfa-894b-ed99dadb1423
payments-gravy-20260307.gz:Mar 6 11:09:47 payments1005 gravy_gateway: 245260402:245260402.1 Calling approvePayment on PSP reference f1479244-7cc2-4c0f-890d-56b21ef06794

fundraising-misc-20260307.gz
Mar 6 11:11:06 : SPCID-1840124194 | (corr_id-gravy-245260402.2) Processing captured Gravy payment with authorization reference '1ee7a916-ce2a-4522-8c7a-f706a5a30be0' and order ID '245260402.2'. | |
Mar 6 11:11:06 : SPCID-1840124194 | (corr_id-gravy-245260402.2) Processing captured Gravy payment with authorization reference 'c393558a-7a08-4cfa-894b-ed99dadb1423' and order ID '245260402.2'. | |
Mar 6 11:14:02 : SPCID-1768573661 | (corr_id-gravy-245260402.1) Processing captured Gravy payment with authorization reference 'f1479244-7cc2-4c0f-890d-56b21ef06794' and order ID '245260402.1'. | |

Change #1255890 had a related patch set uploaded (by Damilare Adedoyin; author: Damilare Adedoyin):

[wikimedia/fundraising/SmashPig@master] WIP: Add idempotency key to Gravy transaction requests

https://gerrit.wikimedia.org/r/1255890

Cstone renamed this task from Donor donated 3 times in 7 seconds with gr4vy google pay and had a duplicate invoice id to Donor donated 3 times in 7 seconds with gravy google pay and had a duplicate invoice id.Mar 20 2026, 3:32 AM
Cstone updated the task description. (Show Details)

Looks like those 3 calls to gravy were in 3 different hits to https://payments.wikimedia.org/api.php , so the root cause of the extra charges seems to be client-side not server side.

The first one, 245260402.1 started at 11:09:13, got an authorization response back at 11:09:15, then logged 'Preparing to run custom filters' and nothing else till 11:09:47 'Finished running custom filters'. Maybe the minfraud query took a really long time?

I think I did get to the bottom of why the next two share an invoice id:

The second request came in at 11:09:18 and got the incremented invoice id 245260402.2. Its auth response came back at 11:09:20. But before that auth reponse came back, the third request had already come in and got the same invoice ID. This happens because we only call incrementSequenceNumber AFTER the auth response comes back. Maybe we should instead do that after building the createPayment parameters but before making the API call?

It looks like for the Gravy google pay form JS we have neither the debounce nor the form disablement that we have for other submit buttons.

Change #1256701 had a related patch set uploaded (by Ejegg; author: Ejegg):

[mediawiki/extensions/DonationInterface@master] Prevent accidental double Google Payments on Gravy

https://gerrit.wikimedia.org/r/1256701

Change #1256701 merged by jenkins-bot:

[mediawiki/extensions/DonationInterface@master] Prevent accidental double Google Payments on Gravy

https://gerrit.wikimedia.org/r/1256701

Change #1259241 had a related patch set uploaded (by Eileen; author: Eileen):

[wikimedia/fundraising/SmashPig@master] Bubble up Adyen Reversal Txn ID

https://gerrit.wikimedia.org/r/1259241

Change #1259241 merged by jenkins-bot:

[wikimedia/fundraising/SmashPig@master] Bubble up Adyen Reversal Txn ID

https://gerrit.wikimedia.org/r/1259241

It's been a month and we've rolled out a couple things to handle this, I'll move this to done and if it happens again we can make a new ticket

XenoRyet set Final Story Points to 4.