Thanks @SHust, I read through the tickets and am pondering if anything is wrong. A few said they accidentally hit donate twice. One said they got 2 emails (which is correct, if they signed up for monthly convert).

Are there any more tickets like this per chance? It's the first time we've run the Monthly Convert on the payments wiki page in these countries, so is it possible this is a natural uptick in people being a little confused?

@AMJohnson do you have anything to add here? If not we can lower the "radar" flag and move on. Thanks, everyone!

TY @SHust! I haven't seen any others so I think it's cool to close this out. Possibly just donor error.

Reopening this one as we are seeing many more of these come through since launching yesterday. Handful of examples:

ZD 1136675 / CID 56104535 (spain / card)
ZD 1136679 / CID 41421400 (spain / card)
ZD 1136760 / CID 56111374 (belgium / card)
ZD 1136773 / CID 41799719 (poland / card)

They all mention a confusing message / believing their first attempt failed.

Can we please look into this as a priority?

It seems like some of them are seeing the "session expired, please try again" error message?

@krobinson have we ever see this in other countries where we've tried MC like Sweden for example? I would assume yes if it's happening across multiple countries ... and about how many total?

@TSkaff - confirmed via ZD 1136760 that is what they are seeing.

We have not seen this before from donors, in Sweden or Italy - or any recent campaign.

I was just digging around for historical info if it helps anyone:

Italy = Post-payment MC Ingenico
Sweden = Post-payment MC Adyen (but the campaign ran at 75% traffic)
France = We didn't have Adyen/Post-payment MC set up yet

So other than Sweden, these bundle countries are the only other places we've run MC Post-payment on Adyen. Maybe that doesn't help anything, but thought I'd add it here just in case.

If there is no other fix, is there any way we could change the message on the error page so it doesn't invite them to try again?

I have initiated an inquiry with Adyen to see if they can determine the extent of the duplicates. Because they are the same email but two separate transactions within a few minutes, I can't get at the duplicate impact from their console. I'm also testing to see if I can duplicate the user experience that would cause this.

So far, I have been unable to reproduce this in incognito testing and with several attempts in each bundle market.

Okay, I was able to reproduce. This was my payments URL: German language, in Austria, 2 € recurring donation

https://payments.wikimedia.org/index.php?title=Special:AdyenCheckoutGateway&appeal=JimmyQuote&country=AT&currency=EUR&payment_method=cc&recurring=1&gateway=adyen&variant=monthlyConvert_011&uselang=de&amount=2&opt_in=1

I filled out info, clicked "Donate" button, nothing happened. Clicked again, got the message (in German)

Your session has expired. Please reload the page and send it again.
Error reference: 121442324.1

However checking Civi my recurring donation has gone through. I used American Express, but it looks like the other reports are from a variety of card brands

I see the following messages in the browser console when the form loads, probably caused by my use of µBlock Origin. Not sure if that might be related.

checkoutshopper-live.adyen.com/checkoutshopper/assets/js/live_VRKTEU4CGRFWJDJ2EEDZXONZQUPOWPFC/fingerprintjs2.js?parentOrigin=https%3A%2F%2Fpayments.wikimedia.org:1          
Failed to load resource: net::ERR_BLOCKED_BY_CLIENT
checkoutshopper-live.adyen.com/checkoutshopper/assets/js/live_VRKTEU4CGRFWJDJ2EEDZXONZQUPOWPFC/fingerprintjs2.js?parentOrigin=https%3A%2F%2Fpayments.wikimedia.org:1          
Failed to load resource: net::ERR_BLOCKED_BY_CLIENT
checkoutshopper-live.adyen.com/checkoutshopper/assets/js/live_VRKTEU4CGRFWJDJ2EEDZXONZQUPOWPFC/fingerprintjs2.js?parentOrigin=https%3A%2F%2Fpayments.wikimedia.org:1          
Failed to load resource: net::ERR_BLOCKED_BY_CLIENT

Okay forget the bit above about µBlock Origin console warnings, I was able to reproduce this again with it disabled.

The donation only goes through on the second click of Donate, the one which causes the error to appear. The first one seems to do literally nothing, but there's no errors in the console

For the record both of these were on Chrome 103, macOS 12.4

I have sent this info over to Adyen for review.

I didn't see anything similar making a one-time donation. Note that all the CIDs referenced were initially recurring donations too.

Noted. Likely why I couldn't reproduce. Mine were all one time with MC after.

If it's helpful below are a few more donor tickets we've received today. Each of the donors accidentally submitted two recurring donations. Haven't seen any screenshots from donors. Both of the donors recurring donations were processed successfully in Adyen. We are manually cancelling and refund the duplicates. In regards to scale I've solved about 13 of these today and I can see that there are more in the queue. Most donors just advise that they "accidentally donated twice" such as in ticket 1136819 and 1136757 but some donors as indicated below provided a bit more info. regarding the experience. Not a lot of technical/device details yet as we haven't heard back from donors yet. Will update should they provide more info.

Escalated to Adyen executives having not heard from Adyen support as of 1:45 Pm Tuesday.

Thanks, all! FR-Tech is digging into this right now, btw...

Update from Adyen is as follows, which I have pushed back on as even if WMF is initiating two separate transactions, why would the user be getting an 'expired page' error. @fr-tech. Can we validate the flow all the same to be sure this isn't on our end. From Adyen: Given that these attempts are coming from webservice user ws@Company.WikimediaFoundation our recommendation would be to review the checkout flow in order to ensure that the integration is not triggering multiple attempts for the same checkout flow. See below one of the examples provided we query the shopperEmail and see two attempts with their own merchant reference and timestamp.

@EMartin Over here in FR-tech land, we have been able to replicate the bug, and we're currently working on figuring out if it's our side or theirs.

Thank you. I am in the process of refunding the duplicates with the payments team.

@Fr-Tech. I just forwarded a thread from Adyen with a question. Can you respond there? They found an error int he log Peter sent: For that checkout page I was able to see an error in our logs:

Error validating client key JSON: {"token":"live_VRKTEU4CGRFWJDJ2EEDZXONZQUPOWPFC"}

This seems to relate to the client authentication key configured for ws@Company.WikimediaFoundation. Could you verify if this is the correct key used in the checkout flow?

https://docs.adyen.com/development-resources/client-side-authentication/migrate-from-origin-key-to-client-key#web-components

@EMartin We think we've actually tracked down the problem and the underlying bug. We're writing up the ticket to fix that now, but we should also have a workaround that lets us bring banners back up in a few minutes.

There's an underlying bug regarding having monthly convert both default and hardcoded in a variant, so the workaround is to take out the monthly convert hardcoding out of the banners. We've talked through that with @HNordeenWMF, and she's going to make the necessary changes to the banner code.

Thanks @XenoRyet. Good to know. I will have Adyen stand down on this. Note to the task: Sandra sent an email to Asjah to have her and I stop refunding as DR wishes to handle this.

Ok, I just finished removing hardcoded Adyen and monthlyConvert_011 for credit card from Bundle banners.

Here is desktop large banner @EMartin are you able to repeat your test donation set-up using this link and see if its fixed?

note from Christine: reminder to do your test in an incognito window!

I still got an error. Is it possible something's caching?

I still am getting errors @HNordeenWMF telling me to contact donor relations. Here it is in German as I used Peter's link and even an alternate and same problem:

@TSkaff yes I think it the variants can be sticky! Try incognito?

I still see the authorization make it Adyen and the transaction is preparing to settle despite the error too. Same problem.... @XenoRyet @HNordeenWMF

@EMartin The link in your screenshot looks like it still has the monthly convert variant in it. I don't think that should be there for the workaround.

I am using incognito when getting these errors fyi

I tried from donate wiki as well with the same problem

I'm still seeing error msgs as well, on Incognito and using the banner w/o the variant. Are other people testing?

Note that Thea is getting errors too, but I see her transactions getting through in Adyen. She THINKs they didn't but they did. Same problem as our users.

Hi @EMartin , @TSkaff did you try without the variant in an incognito window?

I tested and saw it working for me. If you keep seeing issues I can try again in an hour.

yes, all of my testing is in incognito mode and without the variant as well.

Yes, tried Incognito and using the banner w/o the variant. I'm going to give it a little rest time & try again soon.

@Cstone Hi Christine, I was able to get one payment through successfully with a TY page. However I tried again and got this error again:

I get some that go through as well. But the frustrating thing is that this was happening to donors 25% of the time, so when we see one go through, we can't conclude the issue is resolved, no?

To reiterate Peter's findings of earlier today, the error occurs when a donor selects give monthly (not give once). Those are the transactions that seem to intermittently reject to the donor but then they go on to clear in Adyen. I'm still finding this to be a problem. No resolution yet to recurring as I see it. Yes, tested in incognito mode, yes tested with various variants. Same issue.

Ok, let me give a little more context on what we actually found. If the donor naturally picks a recurring donation in a market where we have monthly convert on by default and the banner has the monthly convert variant present, then the donation will proceed to the point where the monthly convert modal should show up, but the modal is invisible or hidden for some reason not yet determined, and the donor doesn't go to the thank you page, but that initial donation is successful.

Then, if they refresh and try again, their session gets killed and they get the "expired session" error message. This attempt doesn't result in a donation.

I believe from there a donor might try a third time, but only perceiving it to be their second, and get another successful donation through at that point, but they still hit the initial bug if they picked recurring again.

We were able to replicate this 100%, and taking the monthly convert variant out of the banner link seemed to resolve it and let the flow continue as intended, but sounds like testing isn't bearing that out. Does the above match other's understanding of what's going on? I want to make sure I'm on the same page as everyone else here.

That's super helpful context Dylan, thank you. So did many folks from fr-tech test/vet it then?

I was going to keep trying a bit more this evening b/c this issue was only appearing 25% of the time, which had me a bit concerned esp when the errors kept appearing ... so I closed my browser, restarted my machine (the old on/off switch technique) and so far have 2 successful transactions. Should I keep trying?

cc @EMartin

@TSkaff I think if you got two successful attempts, then you're getting the flow that fr-tech expects, so probably no reason to keep trying, but I don't think we can use just you to prove it's fixed.

I just tested it three times and wasn't able to replicate, I will try again tomorrow though.

@XenoRyet Thanks for the explanation, but if we have turned off MC presently by shutting down the campaign, how can we test this as being resolved?

I think the way forward is to have some more folks try test recurring donations from the updated banner, with the monthly convert variant hardcoding removed, in a context where we know there's no caching issues. If that goes well, we can feel confident about bringing the campaign back up and monitoring from there.

Just catching up on this - thank you for all of the context.
We will work to refund / cancel duplicates.

As we saw this in the pretest (and I'm guessing the same coding was used for both bundle pretests) could we have a list of duplicate transactions made from those two dates too @Pcoombe / @EMartin ? It feels like we should proactively refund / cancel duplicates, as they will only come back to bite us down the line.

Thanks!

Change 813583 had a related patch set uploaded (by AndyRussG; author: AndyRussG):

[mediawiki/extensions/DonationInterface@master] Check for monthly convert DOM elements in Adyen

https://gerrit.wikimedia.org/r/813583

Hi all! Thanks so so much everyone for the work on this...

Quick update: I've tracked down the full details of what's happening. As described above, the issue should only occur when "variant=monthlyConvert_011" (or the name of another monthly convert variant, of which we have a few) is included in the URL the donor uses when they go to Payments, and the donor chose to make a recurring donation. Please see T312905 for technical details.

I've also made a small patch (not yet deployed) for Adyen that should fully prevent the issue from occurring, even when "variant=monthlyConvert_011" and "recurring=1" are both present on the URL. With that patch, if that happens, the donation will be set as recurring, and no monthly convert dialogue will be shown.

Note that I haven't dug into the Ingenico flow on this, so I can't be sure there isn't a similar bug there, too.

Until the patch with the fix is approved and deployed, it seems the changes to the URLs in banners should do the trick (unless there's some other completely different, unknown issue happening at the same time, which is... ahh never impossible...).

Thanks so much once again!! Giant hug to all...

In T310915#8073934, @TSkaff wrote:

That's super helpful context Dylan, thank you. So did many folks from fr-tech test/vet it then?

I was going to keep trying a bit more this evening b/c this issue was only appearing 25% of the time, which had me a bit concerned esp when the errors kept appearing ... so I closed my browser, restarted my machine (the old on/off switch technique) and so far have 2 successful transactions. Should I keep trying?

To clarify, I don't know the error was happening 25% of the time, only that people were duplicating their donations ~25% of the time. Some people might be abandoning after the first donation on seeing the error (they might also see the TY email before retrying if it comes through quickly enough)

In T310915#8074043, @krobinson wrote:

Just catching up on this - thank you for all of the context.
We will work to refund / cancel duplicates.

As we saw this in the pretest (and I'm guessing the same coding was used for both bundle pretests) could we have a list of duplicate transactions made from those two dates too @Pcoombe / @EMartin ? It feels like we should proactively refund / cancel duplicates, as they will only come back to bite us down the line.

Thanks!

Sure, I'll send a list through later today

In T310915#8073773, @EMartin wrote:

@Fr-Tech. I just forwarded a thread from Adyen with a question. Can you respond there? They found an error int he log Peter sent: For that checkout page I was able to see an error in our logs:

Error validating client key JSON: {"token":"live_VRKTEU4CGRFWJDJ2EEDZXONZQUPOWPFC"}

This seems to relate to the client authentication key configured for ws@Company.WikimediaFoundation. Could you verify if this is the correct key used in the checkout flow?

https://docs.adyen.com/development-resources/client-side-authentication/migrate-from-origin-key-to-client-key#web-components

Just checking, fr-tech was this related (due to the expired session) or is it something else which should be looked into?

Bundle banners went back up at 10:58 UTC without the variant=monthlyConvert_011 parameter, and I don't see any contacts with duplicate recurring donations since then.

Also I tested Ingenico for the same issue and couldn't reproduce it.

Change 813583 merged by jenkins-bot:

[mediawiki/extensions/DonationInterface@master] Check for monthly convert DOM elements in Adyen

https://gerrit.wikimedia.org/r/813583

@Pcoombe, @Ejegg: This task has been open and "Unbreak now" for three weeks, with no updates since July 13th. Any updates / should this remain open? Thanks.