Also GC 5771315413 refund from 3/9 is not in Civi, nor 5391329247 from 3/12 - is something up with the audit process?

We're seeing more recent cases where Ingenico & Civi are diverging. In some cases we manually refunded the most recent of a recurring donation, and it appears as a refund of the initial donation in Civi, and in other cases the refunds are not migrating to Civi at all. This is complicating our followups with donors.

CID 1038946 is the former, CID 24030355 is the latter.

There are two Ingenico refunds that processed in multiple transactions due to DS agent errors:

5499587546 from 2018-03-31 23:38 EUR 20 & EUR 180
6505149333 from 2018-04-10 12:01 EUR 3 & EUR 7

Neither's in Civi yet. Sorry for adding to the wobble pile, and hope they don't complicate the audit process too much. Please let me know if this kind of thing should be a separate task. I've followed up with the agents (& Ingenico's console is somewhat unclear in displaying Req Amount vs Order Amount, tbh) and we'll make sure to only refund in full amounts going forward.

I found the refund that got mapped to the wrong recurring transaction in completed/wx1.000000657020180308.010232.xml.gz--I'm going to keep looking but I'm thinking it doesn't have the full txn id we use--just the order id.

hi @mepps another data point, Ingenico 1670332746 cid=22946716 was one of the proactively canceled recurring donations. We refunded 4 of the 5 donations on April 2nd, but only one of the refunds has migrated to Civi. The first donation of the sequence shows as refunded in Civi, not the subsequent extras. Ingenico 9195751297 is the same case

So, looks like in Civi all the recurring globalcollect refunds are marked incorrectly :(

In reality, we refunded all after the first, but in Civi only the first is marked refunded. So we'll need to undo the refund status for the first in each of those subscriptions, get @mepps's fix out, and re-run the audit files to mark all the subsequent ones refunded.

OK, so one more thing happening here: When the audit script got a refund for e.g. payment 777 installment 5, it was smart enough to look at payment 777 installment 5 and see that it wasn't refunded, then send a message to the refund queue. Unfortunately, the process pulling that message into Civi would refund payment 777 installment 1. Then the next time the audit script ran, it would see that 777 installment 5 still wasn't refunded, and send another message. With all these duplicate messages, the refund queue crept up to 84,000 messages, which is probably why some still haven't gotten into Civi at all.

I've temporarily set the refund queue to run overtime to consume all of these, but we should get that fix out tomorrow!

Change 429097 had a related patch set uploaded (by Ejegg; owner: Mepps):
[wikimedia/fundraising/crm@master] Query for updating contributions mistakenly marked refunded

https://gerrit.wikimedia.org/r/429097

Change 429097 merged by jenkins-bot:
[wikimedia/fundraising/crm@master] Query for updating contributions mistakenly marked refunded

https://gerrit.wikimedia.org/r/429097

Change 431767 had a related patch set uploaded (by Mepps; owner: Mepps):
[wikimedia/fundraising/SmashPig@master] Fetch messages from DamagedDatabase

https://gerrit.wikimedia.org/r/431767

Change 431767 merged by jenkins-bot:
[wikimedia/fundraising/SmashPig@master] Fetch messages from DamagedDatabase

https://gerrit.wikimedia.org/r/431767

Change 432006 had a related patch set uploaded (by Ejegg; owner: Ejegg):
[wikimedia/fundraising/crm@master] Try again to fix refunds

https://gerrit.wikimedia.org/r/432006

Change 432006 merged by jenkins-bot:
[wikimedia/fundraising/crm@master] Try again to fix refunds

https://gerrit.wikimedia.org/r/432006

OK, all the first installments are un-refunded, now to re-run all the audit files for the last couple months.

Change 432032 had a related patch set uploaded (by Ejegg; owner: Ejegg):
[wikimedia/fundraising/SmashPig@master] Fix gateway_parent_id for recurring refunds

https://gerrit.wikimedia.org/r/432032

Argh, unfortunately the re-parsed refunds are all going right back to the original installments. I think that last patch ^^^ will associate them correctly.

Change 432033 had a related patch set uploaded (by Ejegg; owner: Ejegg):
[wikimedia/fundraising/crm@master] Once again reset initial contrib status

https://gerrit.wikimedia.org/r/432033

Change 432032 merged by jenkins-bot:
[wikimedia/fundraising/SmashPig@master] Fix gateway_parent_id for recurring refunds

https://gerrit.wikimedia.org/r/432032

Change 432033 merged by jenkins-bot:
[wikimedia/fundraising/crm@master] Once again reset initial contrib status

https://gerrit.wikimedia.org/r/432033

There seems to still be a problem with associating the recurring payment refunds with the correct installments.

I'm trying to dig into this a bit but struggling to find where the error would be happening now.

Seeing if I can catch it in a test. @Ejegg have we seen this with any other gateways?

no - I think it has to do with the way the old GC integration does recurring. All installments in a series get the same payment ID, but a different 'effort ID'. So we're apparently not finding the right installment with the code we've got now. It SHOULD be possible to recreate in a test, though.

Change 463521 had a related patch set uploaded (by Mepps; owner: Mepps):
[wikimedia/fundraising/crm@master] Trying a test for ingenico refunds with installment number

https://gerrit.wikimedia.org/r/463521

@MBeat33 Do you have any recent examples of where you've seen this? All the older examples seem to have been corrected now.

@mepps I haven't seen any similar cases for refunds. Far as I can tell the recurring refunds are set.

Change 463521 abandoned by Mepps:
Trying a test for ingenico refunds with installment number

https://gerrit.wikimedia.org/r/463521