This is a repeat of the issue in T401324: NL users are reporting "Forbidden" errors when attempting to pay via iDEAL.. The reason for the issue is the same, but the error message is different due to us updating the original "Forbidden" message to the new error message

From the logs, the IP in question is in the Cloudflare subnet that we think includes their WARP proxy service. The IP got blocked after tripping IPVelocityFilter.

Aug 13 09:34:30 payments1007 gravy_gateway: 2xxxxxxxx:2xxxxxxxx.1 IPVelocityFilter: x.x.x.x has 1 hits
Aug 13 09:36:57 payments1007 adyen_gateway: 2xxxxxxxx:2xxxxxxxx.1 IPVelocityFilter: x.x.x.x has 2 hits
Aug 13 09:38:24 payments1007 adyen_gateway: 2xxxxxxxx:2xxxxxxxx.1 IPVelocityFilter: x.x.x.x has 3 hits
Aug 13 09:39:55 payments1007 adyen_gateway: 2xxxxxxxx:2xxxxxxxx.1 IPVelocityFilter: x.x.x.x has 4 hits
Aug 13 09:40:21 payments1007 adyen_gateway: 2xxxxxxxx:2xxxxxxxx.1 IPVelocityFilter: x.x.x.x has 5 hits
Aug 13 09:41:51 payments1007 adyen_gateway: 2xxxxxxxx:2xxxxxxxx.1 IPVelocityFilter: x.x.x.x has 6 hits
Aug 13 09:41:52 payments1007 fail2ban.actions[909264]: NOTICE [payments_fraud_RfT] Ban x.x.x.x

Correction re. logs, it was ~not~ the IPVelocityFilter messages that triggered fail2ban, it was "Redirecting for transaction"

Task name updated to reflect that this issue is happening globally, example from the UK: https://wikimedia.zendesk.com/agent/tickets/1712354, which mentions the Safari Browser.

Reposting original root cause for this from the earlier ticket T401324: NL users are reporting "Forbidden" errors when attempting to pay via iDEAL. that we closed before renaming the error.

@Sandra Moreira-Hust, we've gotten to the bottom of this.

TLDR: This was a network issue on our end where we mistakenly blocked donors from donating due to issues with a third-party network service we use.

Full explanation: Our websites use a service called Cloudflare, which acts as a proxy and a DDOS protection service. Essentially, Cloudflare sits between our servers and the internet. When a donor visits our site, they first connect to a Cloudflare server, which then forwards the request to us. Because Cloudflare has a massive network, multiple users from different locations can appear to be coming from the same Cloudflare-assigned IP address.
To prevent our IP velocity fraud filter from mistakenly blocking multiple donation attempts that appear to be from the same IP, we maintain an allowlist of all official Cloudflare's IP ranges. In this specific case, the donors from the Netherlands and other affected locations were assigned Cloudflare IPs that have not been published and were not on our allowlist. As a result, our system incorrectly flagged and blocked them.

@MBeat33 had a related query on Slack here. After determining the donor has been rejected due to an IP Velocity block. I looked into the related session data and was able to confirm that the user was using Apple's iCloud Private Relay

Upon reading the Apple docs, it looks like we could identify traffic from this service using their published IP lists. This is from their site:

Trust Private Relay connections
All connections that use Private Relay validate that the client is an iPhone, iPad, or Mac and that the customer has a valid iCloud+ subscription. Private Relay enforces several anti-abuse and anti-fraud techniques, such as single-use authentication tokens and rate limiting. This is designed to ensure that only valid Apple devices and accounts in good standing are allowed to use Private Relay. Additionally, the relay IP address will remain stable during a browsing session while a user is interacting with your website.

Traditional fraud detection that relies on IP addresses might need to be adjusted to ensure legitimate users aren’t impacted. Your server can recognize traffic from Private Relay by using the above list of Private Relay IP addresses or by checking how it’s categorized by your geo IP database provider.

They include a list of relay IPs here

Hi @MBeat33 @SHust @KHancock99 @krobinson

About six weeks ago, I mentioned on Slack that we were upgrading our fraud filters to apply more leniency to traffic that comes in via "trusted" proxy services like Apple's iCloud Private Relay.

We've just deployed the updates to support that. Sorry it took so long, we were a bit distracted with the recent fraud spikes, but we got there in the end. We should see a noticeable improvement on this front and a lot fewer blocks due to this issue. We'll monitor it over the next few days to make sure the leniency rules are taking effect. In the meantime, if you still see these reports or anything related, let us know and we'll dig in!

Thanks for your patience on this!

Hi @jgleeson - I wanted to flag that @EBrill-WMF shared there has been a notable spike in donors reporting this issue today after the UK fundraising email deployed. Are you seeing the same thing?

Hi @krobinson, do you have any examples I can take a quick look at?

@krobinson I can only see one IP today that hit the 20-attempt limit, so this isn't the same issue. I think this might be related to other anti-fraud measures, but we'd need an example to dig into. I'm OOO today, but someone will pick this up, so if you could drop an example or two, we could work out what's happened. Thanks

If this is the same issue, donors getting blocked for using shared IPs, then another possible explanation is that those donors are using other "legit" shared services, e.g., Cloudflare's VPN, which we're not granting leniency to. We will be able to confirm this once we get an IP from the samples.

Hi team, here a re few examples from today:

UK - ZD 1765477 - CID 29293097 - received error: 91462bc5ad1cb8d870d5737eb459d4b9 "too many requests"
UK - ZD 1765261 - CID 19694486 - quote from donor: "The error message reads that there are too many bad attempts"
UK - ZD 1765544 - CID 42809961 - quote from donor: "Unable to donate reason given ‘too many requests’"
UK - ZD 1765461 - CID 53046594 - quote from donor: "Hi, I’ve tried to donate £10 plus 40p admin. But website states there are too many requests’"

These are all from today's send E3.