Page MenuHomePhabricator

Add option to early-exit dedupe jobs based on the volume of contributions being processed
Closed, ResolvedPublic1 Estimated Story Points

Description

The dedupe job can run alongside 'normal' volumes of incoming donations. However, when emails go out the backlog spikes up to some thousands. I am adding the following options to the drush command

--threshold=1000
--threshold_period=5

If a threshold is passed in then the job will early-exit if more contributions than the threshold number have been imported in the number of minutes specified by the period (optional, default 5).

If no threshold is passed in then no threshold logic is applied

Event Timeline

Eileenmcnaughton renamed this task from Add option to abort dedupe jobs based on the volume of contributions being processed to Add option to early-exit dedupe jobs based on the volume of contributions being processed.Dec 1 2016, 6:37 AM

This code is currently on staging. I have been testing with

drush civicrm-merge --batch=2 --threshold=1 --threshold_period=6

Note that on staging there are a currently 3 FUTURE contributions - you won't get any recent past ones but you will get these in the threshold_period. This can be confusing