Page MenuHomePhabricator

Adyen duplicate donations after donors see error message - August 2022
Closed, ResolvedPublic

Description

We've seen several tickets where donors think their donations have failed, but two successful donations show in Civi and the Adyen console, with the same invoice reference number. We are unable to replicate, but have found other examples in Civi.

Can we assess the scale of this? We are refunding the donors who reach out to us, but it would be good to know the cause and the scope of the issue. Here are the examples so far:

example numbertransaction dateZD ticketCivi CIDinvoice referencedonor comment
1August 17th, 2022 8:50 AM-51677290127397111.1 dup-1660726268-
1August 17th, 2022 8:49 AM-51677290127397111.1-
2August 17th, 2022 11:56 AM-56592646127415508.1 dup-1660737543-
2August 17th, 2022 11:55 AM-56592646127415508.1-
3August 17th, 2022 5:18 AM-49959468127389131.1 dup-1660713606-
3August 17th, 2022 4:42 AM-49959468127389131.1-
4August 22nd, 2022 9:23 AM115500452352094127572916.1 dup-1661160364“I tried to make a donation by credit card, which was not accepted.”
4August 22nd, 2022 9:15 AM115500452352094127572916.1 dup-1661160364ditto
5August 12th, 2022 8:01 PM11506917816822127227243.1 dup-1660334549“An error in my donation, I thought that my 1st donation of 20 euros had not worked, so I redid 2 times the maneuver. Which means that you withdrew with my card 3 times 20 euros. It's a mistake I don't want and can only donate 20 euros. I hope you can refund me the 40 euros issued by mistake.”
5August 12th, 2022 7:59 PM11506917816822127227243.1ditto
6August 14th, 2022 9:06 AM115479456559471127303876.1“At each test a message appeared indicating a failure...Two days later...bad surprise: 3X 20€ were withdrawn...Could you provide us with an explanation?” >> note: this one of the three was manually captured / settled by staff
6August 14th, 2022 7:36 AM115479456559471127301394.1ditto
6August 17th, 2022 11:29 PM115479456559471127303876.1 dup-1660940947ditto

If I can dig into the ZD tickets for the donors' browser specs + tech details please let me know.

Event Timeline

I had Adyen look at the problem and they are pointing it to our implementation. They suggest adding idem potency to avoid the duplicates. Details as follows:

From Adyen:
Aleyda Davalos (Adyen)

Aug 26, 2022, 20:51 GMT+2

Hello Evelyn,

Thank you for your patience.

In reviewing the PSP's in questions, we can see that these are duplicate payments in a sense that it was charged to the same shopper with the same amount and similar timestamps (see below).

Merchant Reference 1st PSP 2nd PSP 3rd PSP Time btwn 1st -> 2nd auth attempt Notes
127397111.1
F3Q8JF3GS869BV62

L4SWW6QF395CKMG2

	1:09 mins	The PSP's are being authorised without an issue. Our API response happened in a timely manner.

127415508.1
F995CR3T2BK4K472

MWXMNMF3JCCLHK42

	51 seconds	The PSP's are being authorised without an issue. Our API response happened in a timely manner.

127389131.1
VSD4SCRSXW5MPK42

SLKB8TSS2BK4K472

	36 min	The PSP's are being authorised without an issue. Our API response happened in a timely manner.

127572916.1
CXSJWV4KWJFJ8X62

MNGQ2RFZXW5MPK42

	7 min	The PSP's are being authorised without an issue. Our API response happened in a timely manner.

127227243.1
P9X86M2FJS7WTM42

MGJW8G754P972Q42

	50 seconds	The PSP's are being authorised without an issue. Our API response happened in a timely manner.

127303876.1
F7F9VHBTQTLQ4V42

RTTRXJBTQTLQ4V42

BSHFSNHDXXT3MKG2

2 min The PSP's are being authorised without an issue. Our API response happened in a timely manner.
127301394.1
MZFG9LQN38FVMM42

DG7ZRGWDM4H6SP42

	4 min	The PSP's are being authorised without an issue. Our API response happened in a timely manner.

Please be aware, the logs show we responded in a timely manner to the first API requests with an "Authorized" response. Also, the payments triggered 'AUTHORISATION' notifications, which were [accepted] by your server. Further more, we can see the 'duplicate' payments, were initiated by you which leads us to believe this is an issue with your integration.

We suggest implementing API Idempotency, which allows you to retry a request multiple times with it only being executed once. For more information, please review our Idempotency documentation.

Could you confirm verify if there is any retry logic implemented on your end, which could have triggered the duplicate requests?

I've +2ed the patch you submitted @Ejegg, but before moving it to done, I want to dig into some of the examples to see if we can figure out how things went wrong.

Hmm, based on the logs, it looks like the Adyen API might have timed out, which contradicts the "Our API response happened in a timely manner." from Adyen. We could look at extending the timeout period if we confirm this is happening more than we expect?

After some additional digging, I found cases where the duplicate calls happen without requests timing out, so it's not all due to timeouts. Also, the request timeout limit for Adyen is already set to 15 seconds, which is pretty long, so I wouldn't wanna propose we increase that.

Let's move this to done, and hopefully, the idempotent requests patch will prevent any further symptoms, albeit I'm not quite sure of the cause.