Page MenuHomePhabricator

Update the recurring failure/cancellation logic
Open, Needs TriagePublic

Description

This original task was called "Should we build our own autorescue" but renaming it to what that actually means.

In relation to autorescue, which is Adyen's retry logic for credit card recurring payments:

The current one is tied to Adyen, so if we switch card processors on gravy's backend we will lose this

Questions:

  • Does gravy have a non processor specific one (it might be called something else? I couldn't find anything in the docs that wasn't talking about adyen's)
  • They don't as of 2026-03-13 but it is on their roadmap to make one that would be for all payment methods

Now for our own recurring logic:

  • @Lars brought up that our try 3 times and fail process doesn't have to be that way and we haven't updated it in a long time - we could modify that at least to start?

Currently we have three recurring payment flows:

  • Paypal recurrings before Gravy - These are on their own charge schedule managed by Paypal
  • Our recurring retry flow, if the payment fails we try 2 more times then cancel, sending the donor the first recurring failure email
  • Adyen Autorescue - if the recurring Adyen card payment fails with a retryable reason - we tell Adyen to retry this for a configurable amount of days (currently 15) and they choose when to attempt charges, then return the result to us. If it comes back as unrescueable we cancel and send the donor the first recurring failure email.

Things we should change:

  • Have separate retry/cancellation logic per payment method.
  • Easy to start: spread out the attempt days
  • Easy to start: don't cancel on the third failure
  • More complicated: We have a year of failure reasons and if autorescues were sucessful or not, we can look at this data and build our own autorescue system

Event Timeline

AKanji-WMF added subscribers: EMartin, AKanji-WMF.

@Cstone I don't think they do - They have Card Transactions flow but AFAI can tell this only recharges during a checkout flow - https://docs.gr4vy.com/guides/dashboard/flow/card-transactions

For longer term dunning and intelligent retries Adyen Autorescue seems to be it cc: @EMartin

Cstone renamed this task from Should we make our own autorescue to Update the recurring failure/cancellation logic.Mar 13 2026, 6:34 PM
Cstone updated the task description. (Show Details)

could I see that year of data rescue before we meet? @Cstone

In the recurring meeting this morning, we suggested just changing the retry cadence (within a single month) as a first step, since it wouldn't need any additional emails or a 'replacement' flow. No objections were raised. This article suggests retrying on days 3, 7, and 14: https://www.slickerhq.com/resources/blog/optimal-retry-cadence-soft-declines-q3-2025-data-backed-intervals-ai-scheduling

We currently have a scalar variable 'retry_delay_days' and are using the default value of 1. Let's make that into an array

Change #1267104 had a related patch set uploaded (by Ejegg; author: Ejegg):

[wikimedia/fundraising/crm@master] Add setting for arbitrary charge retry cadence

https://gerrit.wikimedia.org/r/1267104

Change #1268299 had a related patch set uploaded (by Ejegg; author: Ejegg):

[wikimedia/fundraising/crm@master] Add setting for minimum days between charges

https://gerrit.wikimedia.org/r/1268299

This is a pretty big change to our process. I'll pull it into the sprint to provide visibility, since it's already being worked on.

Change #1267104 merged by jenkins-bot:

[wikimedia/fundraising/crm@master] Add setting for arbitrary charge retry cadence

https://gerrit.wikimedia.org/r/1267104

Change #1268299 merged by jenkins-bot:

[wikimedia/fundraising/crm@master] Add setting for minimum days between charges

https://gerrit.wikimedia.org/r/1268299

We have deployed all the code necessary to support changing our retry cadence, but the settings are all left at the current logic: 2 retries over 2 days / cancel after 3 failures. Now we just have to decide what change we want and when we want to deploy it.

I have seen a suggestion of 3 retries, on days 3, 7, and 14 after the initial failure.

The data clearly supports a structured approach: the day-3/7/14 framework provides an excellent starting point, but AI-powered dynamic scheduling can deliver significantly better results.

(from https://www.slickerhq.com/resources/blog/optimal-retry-cadence-soft-declines-q3-2025-data-backed-intervals-ai-scheduling)

In the absence of "AI-powered dynamic scheduling", shall we at least try the 3/7/14 retry for now?
@MSuijkerbuijk_WMF @EMartin @RKumar_WMF @AMJohnson @krobinson

@CHudson-WMF we'd really like to measure the improvement we get with this change. Is there any analytics-related consideration of when we should roll it out?

Hi, @Ejegg--I don't think there's necessarily a timing consideration (unless you wanted to do an experiment), but it might take some time to build the supporting models to measure improvement, depending on what you're looking for. Who's the best person to get us up to speed on the exact changes we're planning to make and what improvements we'd expect to see as a result of those changes?

@CHudson-WMF Right now, when a charge fails for a monthly recurring donation, we retry a maximum of 2 times (some failure reasons require us to immediately cancel the donation). We retry 1 day after the initial failure, and if that fails, 1 more time another day later. If both retries fail, we cancel the recurring donation and invite the donor to start a new one.

We are hoping to cancel fewer donations by changing the cadence of these retries. We would like to add one more retry, and change the dates from the current 1 and 2 days after the initial failure, to 3, 7, and 14 days after the initial failure. Later on we may add more communications and a way to update the associated payment method, but first we would like to get an idea how many recurring donations we can save with a simple change to the retry dates.

Considerations might include seasonality of cancel volume (large numbers of sign-ups in November-December mean a sizeable cohort has a card enrollment of the same age on any given date).

Does that answer your questions?

Thanks--so we aren't adding any new communications yet, just changing the retry timing / amount. Is there are good way to distinguish donations eligible for a retry?

Also, when does the cancellation happen now vs. when would it happen with the new retry logic? Is it immediately after the second failure (now) and then updated to immediately after the third failure with the new logic?

This would not affect any recurring contributions on the old paypal flow, that is, where contribution_recur.payment_processor_id in (19,20). Anything cancelled with a 'do not retry' failure reason should then have contribution_recur.cancel_reason set to '(auto) un-retryable card decline reason code', in case you want to exclude those from some analysis.

The cancellation happens immediately after the second failure now, and would be immediately after the third failure in the next iteration.

Gotcha. And are we logging the retry attempts in any way? Or would the only indicator I have that a retry was attempted be that first recurring failure email?

We do add an activity for each failed recurring payment attempt, e.g. this one:

https://civicrm.wikimedia.org/civicrm/activity?atype=172&action=view&reset=1&id=275866588&cid=6925879&context=activity&searchContext=activity

We could probably make those details into a JSON blob with more info, such as 'is_cancel' or 'retry_date'.

The first recurring failure email is only sent after all retries have been exhausted and the contribution_recur status is set to cancelled.

OK @MSuijkerbuijk_WMF @EMartin @RKumar_WMF @AMJohnson @krobinson we are going to roll out the changes to the failure retry cadence later today.

  • Failed charges will be retried up to three times, not just two. At the 4th failure we will cancel and send the usual 'first recurring failure' email.
  • Instead of retrying at 1 and 2 days, we will retry 3, 7, and 14 days after the initial failure.

Thanks, @Ejegg--I'll put in a task in Asana to put together some tracking for this.

Hi, @Ejegg--I've started working on this and I noticed that there are some activities for recurring retries. Do you know under what circumstances the Recurring Processor Retry - Start and Recurring Processor Retry - Success activities are / were used?

@CHudson-WMF that sounds like the Adyen auto-rescue flow, where they retry a charge for us over a couple weeks. We only activate that for recurring installments, and only for those charged directly via Adyen. The 'Start' activity means that a donation failed (for a retry-able reason, e.g. low balance) but has entered Adyen's rescue flow. The 'Success' activity means that they were able to get a successful charge.

Thanks for the background on that.

This would not affect any recurring contributions on the old paypal flow, that is, where contribution_recur.payment_processor_id in (19,20). Anything cancelled with a 'do not retry' failure reason should then have contribution_recur.cancel_reason set to '(auto) un-retryable card decline reason code', in case you want to exclude those from some analysis.

The cancellation happens immediately after the second failure now, and would be immediately after the third failure in the next iteration.

Hey, @Ejegg--just wanted to check-in on these exclusions again. Can you confirm that payment processors 19,20 should be excluded? Looking at the civicrm_payment_processor table both 19 and 20 are labeled as gravy?

Also wanted to see if you had any more guidance on which failure reasons are considered retryable--I can't use the cancel reasons in case something failed and recovered for a retryable reason and then later failed and was canceled for a non-retryable reason.

Oops, sorry, 19, 20 are the IDs in the local development environment - in production paypal IDs are 6, 7

@CHudson-WMF oops forgot to tag you on that last comment

@CHudson-WMF a lot of these questions overlaps with what I was trying to work on that fr-tech dashboard, I think I was trying to graph what elliott wants to track here

I was working on a list of retryable failure reasons before the offsite but I have a half put together chart on that on the recurring fr-tech monitoring page

Change #1294354 had a related patch set uploaded (by Cstone; author: Cstone):

[wikimedia/fundraising/SmashPig@master] Updating the retryable reasons based on https://docs.gr4vy.com/guides/api/error-codes

https://gerrit.wikimedia.org/r/1294354

Change #1294354 merged by jenkins-bot:

[wikimedia/fundraising/SmashPig@master] Update the retryable reasons based on https://docs.gr4vy.com/guides/api/error-codes

https://gerrit.wikimedia.org/r/1294354

Change #1296080 had a related patch set uploaded (by Cstone; author: Cstone):

[wikimedia/fundraising/crm@master] WIP: Add recurring_activity custom fields

https://gerrit.wikimedia.org/r/1296080

Change #1303013 had a related patch set uploaded (by Cstone; author: Cstone):

[wikimedia/fundraising/crm@master] WIP: Add recurring failure activity custom fields

https://gerrit.wikimedia.org/r/1303013

Change #1296080 abandoned by Cstone:

[wikimedia/fundraising/crm@master] WIP: Add recurring_activity custom fields

https://gerrit.wikimedia.org/r/1296080