We wrote a utility a few months back here to backfill missing gravy backend processor data, so we'll see if we can use that here too to fix this data.

@JMando I think you might have unearthed a bigger issue here than I previously anticipated.

I was playing around with a searchkit here  that I added to see how many donations were missing the extra fields we added for gravy, and I adjusted it to cover the period in your query, and I got affected 3882 records!!! I was not expecting that many.

If I flip the condition to show records that correctly include the fields, I get 10,021, so there are a bunch with that data. However, over 25% of transactions are missing it, so I think we have a bug in the code we're relying on to pull in that data. If I remove the date filter, I see a total of 19,296 affected records vs ~80k records, which include the fields, so roughly 20% of all gravy transactions are missing these fields.

For now, I'll make use of the backfill utility we added previously to fix the known broken records, and then I'll dig into the underlying source of the problem.

Thanks for finding this!

@RKumar_WMF @EMartin FYI

Thanks Jack! Thinking of the implications of this I think recon would be impacted in the way we currently undertake it. Sounds like you have this backfill mechanism to correct the nulls? Thanks for looking into it!

Yep @EMartin. I pulled the missing backend data from Gravy on Friday via our backfill script. It took a while due to the number of records, and today I've been importing the updates into CIviCRM in batches of 5000, which should hopefully be wrapped up by the end of today.

In the meantime, I'm digging into why we are losing this data for some transactions.

Looks like the SmashPig code to send donations to the queue from IPNs isn't setting the backend processor, at least for e1b45053-7f95-48c4-aae2-332581f08af9 (mentioned in the linked ticket)

@Ejegg, I was digging into 3D Secure as a possible source of the problem.

jgleeson Ahh, that could fit. When something goes through 3d secure, it's more likely that we will get the IPN before the donor hits payments-wiki and we send another message to the donations queue from there, or that the donor doesn't even click on the redirect to payments-wiki.

Quick update: @JMando, we've backfilled the affected records highlighted in your initial query, along with another ~19k records affected by the same problem.

We're now focusing on fixing the root cause, which we have a pretty good handle on, and then we'll backfill whatever is left to close this gap in the data. Once again, thanks for bringing it to our attention!

@jgleeson thank you! Just confirmed everything looks good in the database from my end.

Change #1145258 had a related patch set uploaded (by Ejegg; author: Ejegg):

[wikimedia/fundraising/SmashPig@master] Add more fields when recording from Gravy IPN

https://gerrit.wikimedia.org/r/1145258

Change #1145279 had a related patch set uploaded (by Ejegg; author: Ejegg):

[mediawiki/extensions/DonationInterface@master] Gravy: send more to pending on redirect

https://gerrit.wikimedia.org/r/1145279

Change #1145258 merged by jenkins-bot:

[wikimedia/fundraising/SmashPig@master] Add more fields when recording from Gravy IPN

https://gerrit.wikimedia.org/r/1145258

Change #1145279 merged by jenkins-bot:

[mediawiki/extensions/DonationInterface@master] Gravy: send more to pending on redirect

https://gerrit.wikimedia.org/r/1145279

OK, we seem to have fixed the root cause of this.

@Ejegg, it looks like the pending transaction resolver is another route through which donations are coming in without the Gravy backend data being populated. I just updated the Gravy Searchkit to show affected donations from today, after confirming this is still happening. The first one I checked came in via the pending transaction resolver.

Change #1146120 had a related patch set uploaded (by Ejegg; author: Ejegg):

[wikimedia/fundraising/crm@master] Add gravy fields to resolved pending transactions

https://gerrit.wikimedia.org/r/1146120

Thanks for finding that @jgleeson ! I think that patch ^^^ should fix it.

Change #1146120 merged by jenkins-bot:

[wikimedia/fundraising/crm@master] Add gravy fields to resolved pending transactions

https://gerrit.wikimedia.org/r/1146120

Change #1146595 had a related patch set uploaded (by Jgleeson; author: Jgleeson):

[wikimedia/fundraising/crm@master] Test adding Gravy fields to resolved pending transactions

https://gerrit.wikimedia.org/r/1146595

Change #1146595 merged by jenkins-bot:

[wikimedia/fundraising/crm@master] Test adding Gravy fields to resolved pending transactions

https://gerrit.wikimedia.org/r/1146595

Looks like the ones that got backfilled got their wmf_contribution_extra.gateway set to Civicrm Import, I updated them back to be gravy

@Cstone I just checked the last batch of backfilled records, and the gateway seems to be correctly set to 'gravy'. Here's the summary of the 5000 I imported last night

Ok I see it @Cstone ! I just kicked off a new batch of backfilled imports, and although the import progress summary screen shows the correct gateway, if I click into one of the records, I can see that the gateway has changed to 'CiviCRM Import', which is really weird as we're not touching that field. Thanks for finding that!

To correct this for the current and future imports, I'll update the mapping configuration to include the gateway as part of the import, which conveniently already contains the correct 'gravy' gateway. This should fix these going forward.

Okay, I think we’ve now blocked all the routes where Gravy donations could come in without backend processor data. I ran the report today and didn’t see any new cases over the weekend.

The only two remaining entries are ones that appear to have come via Gravy, but don’t exist in the Gravy console, so that’s a separate issue to investigate, unrelated to the missing backend processor data.