@Ejegg just reran all the cancelled ipns from before the fix, which all were already cancelled. I'll keep an eye out if any more of these happen

Other situation, donor had monthly that failed May 21 which should have been after the fix, digging into that

The one I looked at was an annual which makes sense that there will be stragglers (for a year I guess?) from that

Hah this came back on May 26th from https://gerrit.wikimedia.org/r/c/wikimedia/fundraising/crm/+/1290092

Update on the slack convo, last week gravy had us add an ideal specific ipn in the adyen console. I added that but we still got a failure today. I messaged them again with the updated info

Paypal turned on the report generation on their side on Friday and we have ones in their ftp from over the weekend now

okay we have a username and pw!

Ah this was due to our bin filter we made in adyen on may 12 as we were getting a bin attack with 414720

We deployed the tiny fix to stop new ones from happening today, I will keep an eye on it.

@CHudson-WMF a lot of these questions overlaps with what I was trying to work on that fr-tech dashboard, I think I was trying to graph what elliott wants to track here

Talked to Cassie at the offsite about metabase alerts and set some of these up to test (emailing once a day)

This was bothering me so I dug into it more but now I am more confused, this is where I got to tonight:

trying to get my account login for this (having 2fa issues)

Right now the transactions are in the adyen audit file but the adyen audit processor doesn't know what to do with them. We just need to update the audit code to handle these:

Ah okay so it wont be in the gravy console because of that validation error. I think we maybe just need to stop sending the processor_contact_id for all recurring converts

@AnnWF While this is the same error we saw with the paypal ones I think something else is going on here. Recurring convert isn't turned on for gravy paypal.

It's been a month and we've rolled out a couple things to handle this, I'll move this to done and if it happens again we can make a new ticket

For the ones that would signify a bug:
Amount of 0
No country in primary address
No primary email