This is interesting, adding more info to the top here. It's not all apple pay and I wasn't able to replicate it myself. We do have app version tied to donations looking to see if we can figure out when it started

Working on getting them in a searchkit here https://civicrm.wikimedia.org/civicrm/admin/search#/edit/6607

Some recent examples showed in an Adyen fraud scan:

Adding one additional example that came in yesterday. Donor in 1758179 / CID 68824702 donated 3 times by mistake via Google Pay from the WP app. Donor notes "I didn't mean to make three different $5.35 donations on 10/19/2025. I only meant to make one. Please cancel and refund two of those donations ($10.70 total)."

The first two transactions were processed via Adyen with the final attempt processed via Gr4vy. As discussed last week (here) we learned that the WP app routes you to both Adyen or Gr4vy depending on the flow you use. Just guessing but maybe they saw an error on the attempts they tried via the "native" flow which processes through Adyen, and on the last attempt they tried the "other payment method" flow which takes you to Gr4vy. An issue with the "native" flow specifically + the app?

We're seeing about 6 per day. Some recent examples

Just flagging a fresh example. Donor in #1768332 / CID 38486846 donated twice via Google Pay in the app by mistake. Donor notes "I have looked at my bank account and noticed that 2 payments for £15 have gone out. I only chose the 1 payment of £15. There was an issue with the page that was hanging/pending when I submitted it. I re-entered the amount which did the same thing again."

Another donor using Apple Pay on the Wikipedia App has also experienced a duplicate donation, which apparently resulted from in-app messaging that the first attempt had not processed.

"I donated this evening using Apple Pay and the app has a message saying the donation was not processed. I tried again with the same outcome. I have been charged twice due to this error. I would appreciate one of the transactions being refunded."

#1772280 | USA | $10.40 | Apple Pay Visa | ADYEN TH8F5W9VPDZCJS69 | iOS | Wikipedia App

I think I was able to replicate this today, but the logs on our ends look good. I'm asking if the apps team have any more info on their end.

Change #1213613 had a related patch set uploaded (by Cstone; author: Cstone):

[mediawiki/extensions/DonationInterface@master] Add more logging

https://gerrit.wikimedia.org/r/1213613

Change #1213613 merged by jenkins-bot:

[mediawiki/extensions/DonationInterface@master] Add more logging

https://gerrit.wikimedia.org/r/1213613

Okay got even more logging in, it looks like every ~100 or so transactions, the capture call takes 30 seconds

Hey @Cstone, we have another donor occurrence of this happening on the android WP App interface.

The donor replied to their receipt and attached the same timeout error screenshot received when donating via WP app on android which reads: 'Cannot connect to Wikipedia. Please try again later':

After I donated, I got the error message (see screenshot). It made an impression that it was not functioning or worse. #1804112 | cid=19100102

Not sure if it is worth mentioning, but these errors did not present in previous app release versions on android.

Adding an update to what has been going on as it's still happening:

• Discussion is ongoing with the apps team on how to handle the auth/capture calls taking longer than the apps timeout
• On our end we have a thread with Adyen into why some calls are taking 30+ seconds instead of the usual 1-2
• I've been watching the duplicates and refunding them as it's been going on

Per adyen's request, I captured traceroutes to the ips behind 7816c1e8e6da0afe-WikimediaFoundation-checkout-live.adyenpayments.com from payments1005:

payments1005:~$ cat adyen_traceroutes_20251219-192257 
traceroute to 135.84.150.188 (135.84.150.188), 30 hops max, 60 byte packets
 1  reth0-1132.pfw1-eqiad.frack.eqiad.wmnet (10.64.40.1)  0.198 ms  0.125 ms  0.093 ms
 2  xe-3-1-7.cr1-eqiad.wikimedia.org (208.80.154.200)  1.749 ms  1.718 ms  1.688 ms
 3  akamai.prolexic.com (206.126.236.219)  0.628 ms  0.661 ms  0.655 ms
 4  a104-86-70-25.deploy.static.akamaitechnologies.com (104.86.70.25)  0.495 ms  0.568 ms  0.537 ms
 5  * * *
 6  ae154-1112.connect1.plxcon-iad1.netarch.akamai.com (23.211.114.210)  1.483 ms  1.321 ms  1.227 ms
 7  a72-52-24-122.deploy.static.akamaitechnologies.com (72.52.24.122)  40.592 ms  40.535 ms  40.481 ms
 8  * * *
 9  875496BC.adyen.com (135.84.150.188)  40.323 ms  40.251 ms  40.412 ms
traceroute to 185.101.198.188 (185.101.198.188), 30 hops max, 60 byte packets
 1  reth0-1132.pfw1-eqiad.frack.eqiad.wmnet (10.64.40.1)  0.217 ms  0.139 ms  0.103 ms
 2  xe-3-1-7.cr1-eqiad.wikimedia.org (208.80.154.200)  0.389 ms  0.356 ms  0.324 ms
 3  * * *
 4  ae2.3613.edge1.Seattle3.net.lumen.tech (4.69.160.73)  64.911 ms  64.907 ms  65.176 ms
 5  ADYEN-B.V.edge1.Seattle3.Level3.net (4.59.233.250)  65.991 ms  65.960 ms  66.005 ms
 6  * * *
 7  B965C6BC.adyen.com (185.101.198.188)  65.667 ms  66.003 ms  66.111 ms
traceroute to 135.84.151.188 (135.84.151.188), 30 hops max, 60 byte packets
 1  reth0-1132.pfw1-eqiad.frack.eqiad.wmnet (10.64.40.1)  0.171 ms  0.095 ms  0.092 ms
 2  xe-3-1-7.cr1-eqiad.wikimedia.org (208.80.154.200)  0.402 ms  0.371 ms  0.338 ms
 3  xe-2-5-3-2.a05.asbnva02.us.bb.gin.ntt.net (192.80.17.185)  0.494 ms  0.461 ms  0.430 ms
 4  ae-3.r26.asbnva02.us.bb.gin.ntt.net (129.250.3.254)  0.470 ms * *
 5  ae-10.r26.snjsca04.us.bb.gin.ntt.net (129.250.6.1)  64.924 ms  64.997 ms  65.067 ms
 6  ae-8.a04.snjsca04.us.bb.gin.ntt.net (129.250.5.209)  64.797 ms  64.799 ms  64.795 ms
 7  xe-0-4-0-1.a04.snjsca04.us.ce.gin.ntt.net (131.103.117.85)  64.893 ms  64.902 ms  64.860 ms
 8  * * *
 9  875497BC.adyen.com (135.84.151.188)  64.897 ms  64.921 ms  64.932 ms

There are 3 traceroutes here because that's how many ips are backing the hostname at this time. The * lines are expected with certain providers and the max RTT time was at or under 66ms.

This is just a one time snapshot but could be run again and from any of the payments hosts.

We are now collecting the mtr data on each of the payments hosts for each IP behind 7816c1e8e6da0afe-WikimediaFoundation-checkout-live.adyenpayments.com. If we encounter another instance of this we can check the timestamped output for any issues.

We have found a routing issue when Adyen includes 62.146.255.205 in the rotation (T413382). The issue occurs when this ip is in the rotation and the connection for the donation gets that IP from the DNS query. It doesn't happen for all donations since the DNS query returns multiple IPs in a random order so it's the luck of the draw for it to get a particular one. We have verified this routing issue is across all the payments hosts and is not isolated to a particular host.

The issue has been spotted and is related to the address sets used in source nat for the payments subnet. While that is being addressed, we are going to add an address set for this new ip/range. I have reached out to Adyen to get information on what IP/CIDR range to use.

IP range received and added to config. Awaiting deployment in T414393.

The firewall update has been deployed and I have verified that mtr provides a proper route to the end address. Please let us know if you see future instances of this.

I have also removed the mtr cron job so we will stop collecting data related to this.

Thanks @Dwisehaupt I'll keep an eye on it