Recurring charge job not getting through that days charges in 24 hours (February 28 and future problem)
Open, Needs TriagePublic
Actions

Assigned To

None

Authored By

	Cstone
	Mar 3 2026, 4:54 AM

Description

This is currently happening due to February only having 28 days , the recurrings set to charge on the 29,30, and 31 are being combined with the March 1 charges and not all finishing on March 1.

As of 2026-03-03 04:46 UTC it has gotten to recurrings started in November 2025.

I dug into this more yesterday, thanks @MBeat33 for bringing it up!

Currently recurring Adyen charges take 2 seconds per charge. The charge job runs twice an hour for 27 minutes each run, limited by time.

So with Adyen we are currently processing 810 charges per run, so 1620 an hour.
Gravy recurrings are at 540 per run, so 1080 an hour.

Using March 6 as an example:
Recurrings scheduled to charge March 6: 23631 (not including paypal as we don't manage those)
Adyen: 16416
Gravy: 6610
Dlocal: 226
Braintree: 379

Looking at Adyen and Gravy:
At 2 seconds per Adyen charge, it takes 10.13 runs or 10.5 hours to finish all the charges
At 3 seconds per Gravy charge, it takes 6.12 runs of 6.5 hours to finish all the charges

If all of the recurrings for March 6 had been moved to Gravy:
At 3 seconds per Gravy charge, it takes 21.3 runs so 21.5 hours

Details

Related Changes in Gerrit:

	Subject	Repo	Branch	Lines +/-
	Add min/max recur_id params to segment charge jobs	wikimedia/fundraising/crm	master	+23 -2
	Run recurring charge jobs in parallel	wikimedia/fundraising/crm	master	+49 -6

Customize query in gerrit

Event Timeline

Cstone created this task.Mar 3 2026, 4:54 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 3 2026, 4:54 AM

KHancock99 added subscribers: MBeat33, KHancock99, AMJohnson, krobinson.Mar 3 2026, 5:01 AM

KHancock99 added a subscriber: SHust.

Cstone updated the task description. (Show Details)Mar 5 2026, 5:44 PM

Change #1249397 had a related patch set uploaded (by Ejegg; author: Ejegg):

[wikimedia/fundraising/crm@master] Run recurring charge jobs in parallel

https://gerrit.wikimedia.org/r/1249397

gerritbot added a project: Patch-For-Review.Mar 9 2026, 8:43 PM

The attached patch will lets run an odds and an evens job in parallel. We could also write logic to run the adyen job in parallel to the gravy job, but if the idea is to move everything to gravy that wouldn't work in the long run.

Cstone moved this task from Ready for Review to Being Reviewed on the Fundraising Tech - Chaos Crew board.Mon, Mar 23, 6:44 PM

Change #1249397 abandoned by Ejegg:

[wikimedia/fundraising/crm@master] Run recurring charge jobs in parallel

Reason:

Modulo is silly - easier to bucket by start_year or just min/max IDs

https://gerrit.wikimedia.org/r/1249397

Maintenance_bot removed a project: Patch-For-Review.Thu, Apr 2, 4:31 PM

Fun way to get the middle ID to partition jobs by IDs

select median(id) over (partition by cancel_date)
from civicrm_contribution_recur
where contribution_status_id in (2, 5, 14, 15)
AND cancel_date IS NULL
and payment_token_id IS NOT NULL
limit 1;

Change #1267224 had a related patch set uploaded (by Ejegg; author: Ejegg):

[wikimedia/fundraising/crm@master] Add min/max recur_id params to segment charge jobs

https://gerrit.wikimedia.org/r/1267224

gerritbot added a project: Patch-For-Review.Thu, Apr 2, 10:03 PM

Change #1267224 merged by jenkins-bot:

[wikimedia/fundraising/crm@master] Add min/max recur_id params to segment charge jobs

https://gerrit.wikimedia.org/r/1267224

Maintenance_bot removed a project: Patch-For-Review.Sat, Apr 4, 4:31 AM

Configuration for review:

https://etherpad.wikimedia.org/p/recurringChargeSplitConfig

We have deployed the split job configuration, with one job running the older 1/2 of recurrings and the other job running the newer half.

Recurring charge job not getting through that days charges in 24 hours (February 28 and future problem)Open, Needs TriagePublicActions

Description

Details

Event Timeline

Recurring charge job not getting through that days charges in 24 hours (February 28 and future problem)
Open, Needs TriagePublic
Actions