@MBeat33 I think we've fixed this. I tested filtering by the order ID in the description over at https://civicrm.wikimedia.org/civicrm/report/wmffraud/fredge and now only see two distinct rows with related transactions. Before the fixes on this ticket, I was seeing dozens of duplicates for the same order ID. Lemme know if this now behaves as you'd expect! Thanks

We discussed these estimates and decided not to record an activity for each login.

Thanks @Cstone. That might have caused the same issue. I think we're good to close this one. It looks like there are scenarios where stuff breaks, and we miss an import, but it doesn't look like there's a silver bullet to fix them, so I'm thinking we can close this and keep an eye on new instances.

Thanks @Ejegg. I'm not sure how the load ordering explains why it's a new problem. We launched Braintree Venmo back in 2022, and these error reports only started happening a few months back. Could you point to where the preload hints are for Adyen? If we can recreate the issue by messing with the ordering, then I guess it makes sense to try.

After digging into this today with @Cstone and @Damilare, we've done the following:

Ok, we've got something thanks to @Damilare. He was also looking into an unrelated Braintree thing, and he noticed that Braintree had an outage right around the time of our first donor experience issues. It's logged here https://www.paypal-status.com/history/eventdetails/93873

That worked! thanks

So there is something going on here, and I'm leaning towards intermittent Braintree connectivity issues, though I still can't replicate it on my end.

This patch updates process-control to allow us to manage job files going forward https://gerrit.wikimedia.org/r/c/wikimedia/fundraising/process-control/+/1215154

From IRC:

Ya, I figured that was the longer-term fix. We could make a ticket for that.

I hopped on the server to check the cache and tried running the same API call from the server; it worked fine.

Ok, this is enough for us to dig back into this. Thanks for all the info @AMJohnson

I've manually updated the affected records with the PII from the Gravy console, so I'll move this one to done.

The working theory here is that both of these transactions failed to be imported the first time around, likely due to database locks during the import run. Usually, when something like this happens, we can recover the transaction via the webhook notifications or the nightly audit. In this case, the webhook processor pulled in the captured donation. However, due to gaps in the PayPal webhook data, the PII was missing, resulting in the CiviCRM contact being created with incomplete data. @Ejegg's related patch plugs that gap so that this shouldn't happen in the future.

Moving this one into review. I think we've got it.

Just to confirm, the fields are still enabled, they just look disabled!

That JS error in the first screenshots is interesting and might give us something to look for. Thanks for sharing!

The first one initially failed due to a db lock error, according to the logs.

I just tried this locally, and I can't reproduce it. I'm on Linux running Google Chrome.

If this is the same issue, donors getting blocked for using shared IPs, then another possible explanation is that those donors are using other "legit" shared services, e.g., Cloudflare's VPN, which we're not granting leniency to. We will be able to confirm this once we get an IP from the samples.

@krobinson I can only see one IP today that hit the 20-attempt limit, so this isn't the same issue. I think this might be related to other anti-fraud measures, but we'd need an example to dig into. I'm OOO today, but someone will pick this up, so if you could drop an example or two, we could work out what's happened. Thanks

Hi @krobinson, do you have any examples I can take a quick look at?

About six weeks ago, I mentioned on Slack that we were upgrading our fraud filters to apply more leniency to traffic that comes in via "trusted" proxy services like Apple's iCloud Private Relay.

Ah, right, so is it done? I can mark it as complete.

Firefox donors can temporarily switch this off on a per-site basis, so in the short term, we could advise them to do the following:

We've got it!

Thanks to that donor's report, I've tracked down the cause of the issue, at least for Firefox users. It's due to the "Enhanced Tracking Protection" feature blocking third-party embedded forms (Gravy's card fields) from accessing cookies and the browser's Local Storage. These folks are essentially setting the anti-tracking / privacy dial up to maximum.

@KHancock99 this is fantastic! I'll see if we can figure out how to replicate it via the environment settings now that we know what we're looking for.

Yeah, @krobinson, I'm definitely leaning towards this being an issue with the Gravy embedded form fields, although it's been a hard one to confirm so far.

Thanks, @AMJohnson, for the new examples. The amazing folks in fr-online gave us access to a paid account for https://www.browserstack.com/ so we can dig into this more and try to figure out what's going on here. Props @spatton and team

Did we get the webhook notifications in these cases?

Gravy has fixed the bug preventing us from successfully charging Amex recurring ApplePay/GPay subscriptions.

Deferral Update: 20 scheduled charges for tonight's run have been moved back a month. This includes nine from last night that were charged and failed.

Unfortunately, I forgot to defer last night's scheduled batch of recurring charges affected by this issue, so we just got some failmail informing us of a batch of 'Invalid cryptogram' payment errors. I'll find the affected recurrings and move from Failing back to In-Progress and then push their next charge day a month out, alongside the batch scheduled to be charged tonight.

I spent some time looking into parallel test runners to potentially help us out here. I found this one https://github.com/paratestphp/paratest and got that running locally. I picked the SmashPig group to start with as it's the area I'm most familiar with.

Deferral Update: 11 scheduled charges for tonight's run have been moved back a month.

I just tried the Windows 10 + Firefox 143 combo and also couldn't replicate it on 237699273.1