Page MenuHomePhabricator

Recent settled Adyen iDEALs not reaching Civi
Closed, ResolvedPublic


Kristie is helping the donor in ZD 749840, who received errors last week as part of the iDEAL bug during the email send. He replied that his subsequent attempts also received errors. When we search by his email address at Adyen, we see the following donations of 20 EUR, all of which are at status Settled. None are in Civi, and they are 4-5 days old now.


Can someone with log access check to see if this is an audit file error, or potentially something related to T248712: Adyen sending back 'paymentMethod=ideal' in returnURL which is not being mapped? We are concerned that this may affect the next NL email send on Thursday.

Event Timeline

MBeat33 renamed this task from Recent settled Adyen iDEALs not reachning Civi to Recent settled Adyen iDEALs not reaching Civi.Apr 28 2020, 4:28 PM
MBeat33 triaged this task as High priority.
MBeat33 added a subscriber: EMartin.

Looking for the first attempt ID in the logs, I see neither a return to payments-wiki nor an IPN message confirming the payment capture (there are logs of the attempt initialization on our side and of an IPN message confirming the authorization). The return and the capture message are the first two places we can send the donation to Civi.

I see an message with the same IP address in a payments-wiki error log (about the time we SHOULD see a return) saying "an upstream response is buffered to a temporary file". @Jgreen, do you know if users generally see errors when that message is logged?

OK, the ratio of those 'buffered' warnings to total payment attempts has been more or less constant over the year to date at about 1 in 3, so there's no spike in that particular problem. Will look at the rest of those attempt IDs and also at iDEAL IPNs in general to see if we're getting the notifications we expect.

Change 593021 had a related patch set uploaded (by Ejegg; owner: Ejegg):
[wikimedia/fundraising/SmashPig@master] Adyen iDEAL: record payment on AUTH message

Change 593021 merged by jenkins-bot:
[wikimedia/fundraising/SmashPig@master] Adyen iDEAL: record payment on AUTH message

Thanks, @Ejegg - do you know what the 'real' status of these donations is? Should we refund three of them if one will settle into Civi?

@MBeat33 if they're at 'settled' in the console, I'd consider that definitive and that we should refund all but one.

I think they should eventually make it to Civi via the audit file. We only get that weekly from Adyen, though.

Change 593072 had a related patch set uploaded (by Ejegg; owner: Ejegg):
[mediawiki/extensions/DonationInterface@master] Debug logging for Adyen resultswitcher

Change 593072 merged by jenkins-bot:
[mediawiki/extensions/DonationInterface@master] Debug logging for Adyen resultswitcher

Today we dug into the logs and found, for 2020-04-28, 43 iDEAL transactions that Adyen considered to have been successful ("settled" or "sent for settle") but for which the users did not reach the Thank You page.

To do this, we downloaded a CSV of Adyen transactions, which contained data up to 7:10 UTC. Interestingly, there were a lot of successful-but-thank-you-less iDEAL transactions throughout the UTC night, but of the 27 successful and thank-you-ful iDEAL ones in the file, for 2020-04-28, nearly all were between 6:30 and 7:10 UTC.

We could repeat the same comparison tomorrow to see if the issue persists. I can also provide a file with the thank-you-less (thankless?) transactions found, in case that's useful!

@AndyRussG thank you! Please do send me the list of the donors whose transactions have settled but who haven't received TYs, as some of them may have changed methods to donate and we can follow up with them generally.

We could repeat the same comparison tomorrow to see if the issue persists.

nearly all were between 6:30 and 7:10 UTC.

If the patches deployed have not eliminated this, please do repeat the comparison. I am concerned that this is a donor-facing error, and even a 40 minute recurrence of this error would generate a lot of work if it falls right after an email send like the one scheduled for tomorrow.

Donors getting error messages for successful donations, and not receiving TY emails for up to a week may cause a lot of donors to:

  • switch to PayPal
  • send a check
  • be confused into thinking we've taken additional funds when they get a TY email a week later.

If you can provide an estimate of how likely this is to happen again it would be helpful

MBeat33 claimed this task.

After checking in with fr-tech just now, this looks not be an issue during tomorrow's send or going forward. The affected donors from before yesterday's patch will get their TY emails with the weekly audit, and we can follow up with them as needed. Thanks to all who helped resolve this one!

Hi @MBeat33! Yes, it makes sense to go ahead with the send, now that things will also go into civi for donors who don't reach the TY page for whatever reason.

Also, I double-checked my review of the logs from yesterday, and it looks like I was mistaken about successful donors who did not get a thank you page. (I forgot to take into account the mismatch in time ranges from two files I was comparing.)

Looking at logs from 04-29, including entries from the new debug instrumentation that @Ejegg added, things seem to match up! There are still some technical details related to log contents and searching that I'd like to dig into a bit further, but we can do that in another space, I think...

Thanks, all, for all your work on this!

MBeat33 reopened this task as Open.EditedMay 1 2020, 4:32 AM

Kris H found another one. Trxn 80213557.1 was settled and captured at Adyen on 2020-04-25 13:36:34 PDT but is still not in Civi. cid=20454734 is the only donor record with that email address - can you remind me when the patch was applied? What's the date after which we wouldn't expect to see any more stragglers, @AndyRussG ?

04/28 settled

04/28 settled

We are continuing to see bulk email complaints from donors whose recent contributions via iDEAL/Adyen have not reached Civi. Kris just processed about a dozen, of which these are a sample:

751286cid=2633417380240461.104/29 settled
751485cid=2627187180203245.104/25 settled
751397cid=1104892080256771.104/29 settled

Can we tell why these are continuing to not reach Civi in a timely-enough way? It would be great to patch this before fundraising email3 is sent.

Hi @MBeat33 , the fix to the Adyen iDeal payments was deployed on the 28th at 21:55 UTC.

The Adyen audit file with the missing ones from before the fix will come in tonight.

Thank you @Cstone Do you know if the two examples from the 29th should have been prevented by the fix from the 28th?

@MBeat33 I looked at those in the Adyen console, 80240461.1 was made on the 27th and 80256771.1 on the 28th before the patch went out.

Thanks, @Cstone, and sorry for my mistake on the dates for those.