Looking into this, it's not every single dlocal recurring but it's a good amount of them.

Looks like it started September 22.

This issue is not specific to Gravy, as it's also impacting our direct dLocal recurring charges. We've seen around 100 total failures across both integrations in the last 24 hours, and the vast majority of these affected subscriptions are coming from Mexico.

I will put together some examples and contact the dLocal team on Slack to start an investigation on their end.

I've reached out to dLocal here on Slack

Note: When digging into this, I noticed that Gravy is sending over a placeholder tax ID / Document ID of '1111111111' with all Mexico recurring charges, which doesn't seem to be related to this particular problem, but nevertheless, it looks strange. It's likely related to document ID waivers. We are not doing this for our direct integration recurring charges. Examples below:

Gravy-dLocal recurring charge failure:

• T-648-x1kdo655-quo6op24kt5p1d-tl0srn11ieuc

Direct recurring failure:

• T-648-x1kdpj2l-r7mnkhash52rl0-de2jsuuodnfo

@jgleeson Are these subscriptions just failing or actually cancelling? If cancelling, once we've figured this out, could we get a count so we can figure out next steps? Thanks!

@krobinson, the subscriptions will be cancelled after three failed attempts. I expect there will be a good number that have been cancelled.

I'm digging into the numbers, sifting out the suspected unusual instances of this error from the generally expected failures with the same reason. The related error, Invalid security code, occurs rarely for recurring charges, as indicated by the logs. It looks like this became a problem around the 22nd-23rd of last month, based on some raw numbers pulled just now:

Recurring charge error: Invalid security code.

  1 2025-09-05
  1 2025-09-06
  1 2025-09-07
 13 2025-09-08
  4 2025-09-09
  9 2025-09-10
  1 2025-09-11
  2 2025-09-12
  1 2025-09-13
  1 2025-09-14
  1 2025-09-16
  2 2025-09-17
  1 2025-09-18
  7 2025-09-22
 64 2025-09-23
 98 2025-09-24
144 2025-09-25
215 2025-09-26
232 2025-09-27
215 2025-09-28
140 2025-09-29
142 2025-09-30

Change #1193118 had a related patch set uploaded (by Jgleeson; author: Jgleeson):

[wikimedia/fundraising/crm@master] Temporarily skip dLocal MX recurring charges

https://gerrit.wikimedia.org/r/1193118

@jgleeson Skipping them for a while seems like a good first step, but maybe we don't need a code change to do that - how about just moving their next_sched_contribution_date up a few days?

It looks like Gravy is sending the recurring charge differently than we do - if I check on the API call log for this txn:

https://wikimedia.gr4vy.app/merchants/default/transactions/43bcdec5-cd6c-4714-925b-476ef7f5e31f/overview

It shows Gravy sending the card number and expiration date with the second recurring installment.

{
  "amount": 260,
  "currency": "MXN",
  "country": "MX",
  "payer": {
    "name": "<redacted>",
    "email": "<redacted>",
    "document": "<redacted>",
    "ip": "<redacted>",
    "address": {}
  },
  "order_id": "23onaYgYaTjpGAqBrP9zRv",
  "notification_url": "https://api.wikimedia.gr4vy.app/i/miWeE_Y0QUWqvloq8dKmlWRsb2NhbC1jYXJk/94ns0L3xRq5eDgJKL_TXNQ89H0rCxFTyQSAEto5UkK8",
  "callback_url": "https://api.wikimedia.gr4vy.app/transactions/43bcdec5-cd6c-4714-925b-476ef7f5e31f/complete?gr4vy_store=false",
  "additional_risk_data": {
    "submerchant": {
      "name": "••••••••••••••••••••"
    }
  },
  "payment_method_id": "CARD",
  "payment_method_flow": "DIRECT",
  "card": {
    "capture": false,
    "descriptor": "Wikimedia Foundation",
    "save": false,
    "verify": false,
    "holder_name": "<redacted>",
    "expiration_month": <redacted>,
    "expiration_year": <redacted>,
    "number": "<redacted>",
    "stored_credential_type": "SUBSCRIPTION",
    "stored_credential_usage": "USED"
  }
}

By contrast, in our direct integration we send

				"payment_method": {
					"method" => "id",
					"id" => $params['recurring_payment_token']
				},
				"payment_source": "recurring",
				"is_subsequent_payment": true,
				"merchant_initiated": true,

Any idea why Gravy is sending them this way?

In T406090#11237044, @Ejegg wrote:

@jgleeson Skipping them for a while seems like a good first step, but maybe we don't need a code change to do that - how about just moving their next_sched_contribution_date up a few days?

Yeah, that's a good idea. @Cstone suggested a Searchkit, so I'm just writing some SQL now to get an idea of numbers.

So the API call they're making seems pretty much in line with dlocal's docs for merchant initiated transactions:
https://docs.dlocal.com/docs/merchant-initiated-transactions
I guess that points to a bug at dlocal's side being more likely.

Unrelated to the CVV fails, it looks like we should ask them to start saving and sending the card.network_payment_reference - the initial Network transaction ID. That's supposed to make auth rates higher for subscriptions payments.

Here are a few SearchKits.

• dLocal MX recurring subscriptions with the status of Failing (113)
• dlocal MX recurring subscriptions cancelled in the last 9 days (322)
• dlocal MX recurrings that will be charged tonight (and fail) (87)

It looks like we can send the initial network txn ID with the API call to Gravy, but their docs suggest that they already store and re-send that if we don't:
https://docs.gr4vy.com/reference/transactions/new-transaction#body-previous-scheme-transaction-id

If not provided, and a qualifying customer initiated transaction has been previously made with the stored payment method, then Gr4vy will populate this value with the identifier returned for that transaction

Change #1193118 abandoned by Jgleeson:

[wikimedia/fundraising/crm@master] Temporarily skip dLocal MX recurring charges

https://gerrit.wikimedia.org/r/1193118

Temporary Mitigation Plan for dLocal MX Subscriptions

Until dLocal resolves the current issue, we will reschedule the nightly dLocal MX recurring charges one month in advance, effectively skipping the payment for the current month.

We've created a dedicated SearchKit here to display the scheduled daily charge batches, allowing us to mass-update all records in one place and defer their next scheduled charge date.

The only gotcha is the weekend. Since charges run on Saturdays and Sundays, we will need to perform the deferral process on each Friday to reschedule the full weekend's worth of charges for the following month. I've just updated this weekend's batch.

Dlocal have advised today that they have migrated MX to another processor and asked us to now retry the billing attempts. @jgleeson can we retry current month to assess viability without derailing your plan above?

Looks like this is fixed now, dlcoal did a test that went through successfully and I tested two that were going to charge tonight, also successful

Christine was able to confirm that recurrings are now resumed via Mexico since Dlocal made the switch of processors. All has returned to normal in Mexico. @MSuijkerbuijk_WMF for awareness.

Additionally, we have rescheduled the recurrings that were temporarily pushed forward a month, to charge today.

Thanks