Update: I can see a bunch of related failing charges all marked with "Error: 1000014". I've reached out to Gravy here for more information on this error.

Update: we're actually assigning the error code 1000014 here.
My current thinking is that we're unable to make the recurring recharge payments in this scenario due to invalid or missing fields saved on our side. I'm trying to confirm, and then if so, figure out why.

This issue appears to be affecting all Gravy JPY Apple Pay and Google Pay recurring subscriptions. We're temporarily redirecting this traffic directly to Adyen until we figure out the issue.

@EMartin @RKumar_WMF FYI

Change #1160194 had a related patch set uploaded (by Ejegg; author: Ejegg):

[wikimedia/fundraising/crm@master] Log all errors from failed recurring payment attempts

https://gerrit.wikimedia.org/r/1160194

Change #1160194 merged by jenkins-bot:

[wikimedia/fundraising/crm@master] Log all errors from failed recurring payment attempts

https://gerrit.wikimedia.org/r/1160194

We've found the problem and are in the process of testing the fix. Since it was entirely at our end, we would be able to reactivate those donors and start charging them again, if it weren't for the fact that we've already sent out the failure notices.

Yep @Ejegg I was thinking about that option earlier with @krobinson on the Slack thread here. I'm also thinking maybe we should no longer cancel subscriptions due to validation failures on our side, as a follow-up. Lots of learnings on this one!

@krobinson. The underlying issue here was related to the same bug we fixed in T393939: Japan credit card processing error back in May, but unfortunately, that fix was also needed over in CiviCRM and we didn't realise at the time :(

It looks like the recurring bug we found yesterday has a wider reach than just Japan. We're now seeing it in other countries like Brazil, Colombia, and South Africa. The core issue is a missing fiscal number during recurring charges. Since this is popping up in Brazil and Colombia, I suspect we might not be collecting this number during Google Pay and Apple Pay recurring sign-ups. If that's the case, it will block all subsequent charges because the fiscal number is a required field for the API calls.

After digging into this, the issue is becoming clear.

Using Brazil as an example. For standard card payments routed to dLocal, a fiscal number or tax ID is a mandatory field. But for Google Pay card payments, these are sent to Adyen, and crucially, Adyen doesn't require a fiscal number to process the transaction.

The fundamental problem lies in SmashPig. It's making the decision to require the fiscal number based solely on the country, instead of considering both the country and the backend processor. Our current configuration is therefore flawed: the system detects Brazil and enforces the fiscal number requirement, but for Google Pay payments in Brazil through Adyen, that number isn't necessary or available to begin with. This results in repeated validation failures, which then sadly lead to the subscription being cancelled. We have already discussed this being a potential issue, but unfortunately, it was already causing problems at the time, and this issue has been ongoing since we started sending all App payment traffic to Gravy.

For now, let's remove the fiscal number as a requirement for recurring charges. The idea here is that for initial payments, if the fiscal number is needed, it will be captured and stored. Then, for all subsequent recurring charges, we'll only include it in the API request if it was indeed provided initially; otherwise, we'll skip it, leaving out any related validation.

I'll open a new ticket for this update as it's outside the scope of this one. Ticket is here

Change #1163806 had a related patch set uploaded (by Ejegg; author: Ejegg):

[wikimedia/fundraising/crm@master] Use match not switch for getPaymentMethod

https://gerrit.wikimedia.org/r/1163806

Change #1163807 had a related patch set uploaded (by Ejegg; author: Ejegg):

[wikimedia/fundraising/crm@master] Add apple / google to getPaymentMethod

https://gerrit.wikimedia.org/r/1163807

Change #1163810 had a related patch set uploaded (by Ejegg; author: Ejegg):

[wikimedia/fundraising/SmashPig@master] Allow Adyen mobile payment classes to charge recurring

https://gerrit.wikimedia.org/r/1163810

Update: We have added alerting on the recurring charge job so that fr-tech is aware of any failures of this type as soon as they happen.

That's helped us track down a couple more countries that needed this same fix, and also find a few more cases that cause these failures (mostly missing personal data, e.g. from Trustly - see T395375: some Gravy contribs missing PII (PayPal, trustly)).

We've got a list of 900+ cancelled contribution recur IDs that we can use to facilitate communication. @AMJohnson / @KHancock99 can you remind me what donor info you need to be able to reach out to them?

Change #1163810 merged by jenkins-bot:

[wikimedia/fundraising/SmashPig@master] Allow Adyen mobile payment classes to charge recurring

https://gerrit.wikimedia.org/r/1163810

Change #1163806 merged by jenkins-bot:

[wikimedia/fundraising/crm@master] Use match not switch for getPaymentMethod

https://gerrit.wikimedia.org/r/1163806

Change #1163807 merged by jenkins-bot:

[wikimedia/fundraising/crm@master] Add apple / google to getPaymentMethod

https://gerrit.wikimedia.org/r/1163807

@AMJohnson and @KHancock99 I've shared a spreadsheet with a list of the affected contacts

Much appreciated, thank you @Ejegg.

@Ejegg I'll be filing a Phab ticket for the email send, and wanted to check, has the list you shared been reviewed to remove any donors who have since reinstated their recurring donation? Thank you!