Page MenuHomePhabricator

Civi: January 2020 Ingenico recurrings processing in duplicate?
Closed, ResolvedPublic

Description

We have two donors (so far) very upset that their Ingenico recurring donation has processed twice this month. This is donor-facing and will be big trouble if it scales, so I'm assigning UBN status, but feel free to de-escalate.

Zendesk 724937
cid=35942971
only one recurring donation in Civi, but two separate processings in January:
RECURRING INGENICO 000000657040047429460000200001
and
RECURRING INGENICO 000000657040046559360000200001

Zendesk 724953
cid=14014838
only one recurring donation in Civi, but two separate processings in January:
RECURRING INGENICO 000000657040047432940000200001
and
RECURRING INGENICO 000000657040046562320000200001

Kris noticed that in both cases the proper and the duplicate charges processed on the 1st and the 9th, respectively.

Event Timeline

MBeat33 triaged this task as Unbreak Now! priority.Jan 21 2020, 11:00 PM

Kris found a third example:
cid=8506413 Zendesk 725434

Please let me know when the 429 refunds have been batch refunded.

Hi @MBeat33. @AndyRussG (along with @Cstone and @Eileenmcnaughton) exported a file with the problems and ran the batch refund.

The file is on the fundraising fileserver at Tech/Ingenico/T243356.csv, which should have enough detail to send a communication if you want to.

It turns out there were a few PayPal donations among the people with 2 recurring charges in January. The total number of Ingenico donors we charged twice in January is 424. We haven't looked further into the PayPal donations, but they might have to do with the longer wait between failure retries in PayPal's scheduling system.

Just a note - the refund script DOES NOT update the status in Civi, it just makes the refunds at the console. We'll get the status updated in Civi in a day when the next audit file comes in.

@Cstone and I spot checked a couple refunds at the console and they seem to be in status 800 (requested).

@Cstone also figured out the reason for this error - it was related to an incomplete data fix that we applied on December 9th. That fix corrected the cycle_day for a bunch of recurring donations where it was incorrectly set to '1' but didn't fix the next scheduled charge date, leaving it at 2020-01-01. So after the incorrect charge on Jan 1, they were charged on their correct cycle day of Jan 9th. The NEXT scheduled charge dates are good, meaning this won't happen again in February.

We also made a phab ticket for an additional safeguard against multiple charges less than 27 days apart: T240140

Thanks for the comprehensive and speedy followup, @Ejegg and all.

erm shouldn't this still be in the sprint - not sure why the tag was removed?

Here are the queries we used for this:

  • P10261 Find donors who got charged too soon.
  • P10262 Generate CSV for batch refund script.
  • P10290 Check no one will get double-charged.