Looking for the first attempt ID in the logs, I see neither a return to payments-wiki nor an IPN message confirming the payment capture (there are logs of the attempt initialization on our side and of an IPN message confirming the authorization). The return and the capture message are the first two places we can send the donation to Civi.

I see an message with the same IP address in a payments-wiki error log (about the time we SHOULD see a return) saying "an upstream response is buffered to a temporary file". @Jgreen, do you know if users generally see errors when that message is logged?

OK, the ratio of those 'buffered' warnings to total payment attempts has been more or less constant over the year to date at about 1 in 3, so there's no spike in that particular problem. Will look at the rest of those attempt IDs and also at iDEAL IPNs in general to see if we're getting the notifications we expect.

Change 593021 had a related patch set uploaded (by Ejegg; owner: Ejegg):
[wikimedia/fundraising/SmashPig@master] Adyen iDEAL: record payment on AUTH message

https://gerrit.wikimedia.org/r/593021

Change 593021 merged by jenkins-bot:
[wikimedia/fundraising/SmashPig@master] Adyen iDEAL: record payment on AUTH message

https://gerrit.wikimedia.org/r/593021

Thanks, @Ejegg - do you know what the 'real' status of these donations is? Should we refund three of them if one will settle into Civi?

@MBeat33 if they're at 'settled' in the console, I'd consider that definitive and that we should refund all but one.

I think they should eventually make it to Civi via the audit file. We only get that weekly from Adyen, though.

Great, thank you.

Change 593072 had a related patch set uploaded (by Ejegg; owner: Ejegg):
[mediawiki/extensions/DonationInterface@master] Debug logging for Adyen resultswitcher

https://gerrit.wikimedia.org/r/593072

Change 593072 merged by jenkins-bot:
[mediawiki/extensions/DonationInterface@master] Debug logging for Adyen resultswitcher

https://gerrit.wikimedia.org/r/593072

Today we dug into the logs and found, for 2020-04-28, 43 iDEAL transactions that Adyen considered to have been successful ("settled" or "sent for settle") but for which the users did not reach the Thank You page.

To do this, we downloaded a CSV of Adyen transactions, which contained data up to 7:10 UTC. Interestingly, there were a lot of successful-but-thank-you-less iDEAL transactions throughout the UTC night, but of the 27 successful and thank-you-ful iDEAL ones in the file, for 2020-04-28, nearly all were between 6:30 and 7:10 UTC.

We could repeat the same comparison tomorrow to see if the issue persists. I can also provide a file with the thank-you-less (thankless?) transactions found, in case that's useful!

@AndyRussG thank you! Please do send me the list of the donors whose transactions have settled but who haven't received TYs, as some of them may have changed methods to donate and we can follow up with them generally.

We could repeat the same comparison tomorrow to see if the issue persists.

nearly all were between 6:30 and 7:10 UTC.

If the patches deployed have not eliminated this, please do repeat the comparison. I am concerned that this is a donor-facing error, and even a 40 minute recurrence of this error would generate a lot of work if it falls right after an email send like the one scheduled for tomorrow.

Donors getting error messages for successful donations, and not receiving TY emails for up to a week may cause a lot of donors to:

• switch to PayPal
• send a check
• be confused into thinking we've taken additional funds when they get a TY email a week later.

If you can provide an estimate of how likely this is to happen again it would be helpful

Hi @MBeat33! Yes, it makes sense to go ahead with the send, now that things will also go into civi for donors who don't reach the TY page for whatever reason.

Also, I double-checked my review of the logs from yesterday, and it looks like I was mistaken about successful donors who did not get a thank you page. (I forgot to take into account the mismatch in time ranges from two files I was comparing.)

Looking at logs from 04-29, including entries from the new debug instrumentation that @Ejegg added, things seem to match up! There are still some technical details related to log contents and searching that I'd like to dig into a bit further, but we can do that in another space, I think...

Thanks, all, for all your work on this!

We are continuing to see bulk email complaints from donors whose recent contributions via iDEAL/Adyen have not reached Civi. Kris just processed about a dozen, of which these are a sample:

Can we tell why these are continuing to not reach Civi in a timely-enough way? It would be great to patch this before fundraising email3 is sent.

Hi @MBeat33 , the fix to the Adyen iDeal payments was deployed on the 28th at 21:55 UTC.

The Adyen audit file with the missing ones from before the fix will come in tonight.

Thank you @Cstone Do you know if the two examples from the 29th should have been prevented by the fix from the 28th?

@MBeat33 I looked at those in the Adyen console, 80240461.1 was made on the 27th and 80256771.1 on the 28th before the patch went out.