@Ejegg. Awesome! Mark this down in history. Thank you Elliott.

Great! Thanks @Ejegg !

@Ejegg. @krobinson As expected, Finance wants to stay with weekly settlement. I did inquire with Adyen and they advise the following as it relates to the manually settled activity. Can we add this report to the mix to get these settled faster? Would this be a big undertaking? Should I put it in a separate Phab request? It seems like something we need:

Hi @Ejegg Thanks. I will check with Finance. I doubt they will want to increase the frequency since Ingenico has been at weekly but it's a good time to ask. Assume the process was different for Ingenico so we don't see the lags? I'll let you know what I find out from Finance.

+ 1 @krobinson We need to put Adyen on the same track that Ingenico was to avoid these lags that could have bigger impact when we are operating at scale.

Successful GPay test transactions by market.
MYR -

@fr-tech. I was looking at Kristie's recap of donor complaints about this topic. I had a thought about recurring around iDeal since most of the complaints surround iDeal in Kristie's examples: For iDeal, we do the initial transaction on iDeal and the subsequent transaction on direct debit. Does that have any bearing on this matter, i.e., it is unlike credit card and involves two distinct products. Should the initial transaction on iDeal always be one time and recurring reserved for the on going direct debit transactions? As a practice we should never actually send recurring on iDeal itself because it isn't supported. We are using direct debit to satisfy the recurring flow portion of this.

From Adyen:
I had a look at the different between the en-US and en-ES configuration and what stood out was the difference n the paymentMethodsResponse which is null on the en-ES site. I was able to resolve the 'This merchant is not enabled for Google Pay’ error by passing in the paymentMethodsResponse from the en-US site in the en-ES site.

@fr-tech. I processed a Gpay transaction yesterday and it didn't settle yet.

Note to Phab on this issue: Discussion with Adyen and FRTech today. Their advice is to send the parameters to perform 3DS and Adyen will selectively invoke 3DS on the Carte Bancaire network if it is required by issuer. We DID start sending the parameters as of 25 July but still saw declines on Carte Bancaire. Action: Matt, our Account Manager to circle back with Adyen product or Carte Bancaire to explore causes further. Adyen confirms that no other merchant is seeing the extent of the declines that we were so something is still awry. For now we can process on VI/MC and the CB bugged only cards will process through CB, so no lost transactions. Not currently mission critical, but needs sorting. Evelyn to pursue with Matt in weekly calls. On the Payments Adyen tracker and I will update here if I learn anything further.

@HNordeenWMF No, this was corrected by FRTech. The remaining issue surrounds Spain and FRTech is working with Google now to sort that one out.

@greg @TSkaff @dkozlowski Note to Greg, Dylan and Thea that the App team will need dev support to get Apple Pay live in the App. Can we get this in the priority queue when Olga gets to that point? @OTichonova Hi Olga, these are the folks than can prioritize what you need from Fundraising Tech to deploy.

@HNordeenWMF Hi Haley, I ran a quick test on these links above. They look good. I'll ping you in Asana to see if we can get these in the banners. They look good!!

Can we enable Gpay in these markets while we continue to unblock Spain?

Can I help at all?

Associated thread: Gpay rollout errors

@Ejegg Hi Elliott, Adyen is asking if we are filtering by markets. Can we respond? From Adyen: Everything from Adyen's side looks to be correct. The only factor here is when changing the country code to ES, the following error is displayed.
DEVELOPER_ERROR in loadPaymentData: You have not registered to use this API. Double check your Android app or web domain registration.
Are you limiting the countries you accept GooglePay in a regex or perhaps within Google business manager? Also contacting Google to double check the domain registration - even though it looks to be the same - isn't the issue on their side.

We had 3DS both active and inactive on the Carte Bancaire network. Both showed high failures. We moved to VI/MC network and we see 95% approvals now. I note in a spot check of multiple transactions that 3DS does not seem to be on (Jack please confirm). We are seeing solid results using the VI/MC network in France now. I'm still working with Adyen to understand what the challenge is w/ CB's network.

Hi Jack, 3DS is currently not enabled on VI/MC in France from what I'm seeing in the console, correct? If so, I say we leave it as is. We are seeing high approval rates on those networks with our current config.

Note: This is still an issue as we do not know what was causing the CarteBancaire declines on that network. We have rerouted traffic to process on the VI/MC network FIRST but that doesn't solve the issue where we saw 30+% declines for authentication on CarteBancaire. It could be that CB network has more rigorous fraud rules and is turning more transactions away. Adyen is saying our declines on CB were legitimate, however, when processed as VI/MC we aren't seeing the declines. I'm waiting on more info from Adyen on this. I did read on various Payment industry blogs to try the VI/MC first approach and that seems to be working. There may not be anything to fix, just issuers in France being more stringent. I'll post what I learn here.

Yes, the results are looking promising so far using Vi/MC. We'll be monitoring it over the weekend.

I have confirmed that Adyen is retrying on the VI/MC networks. Watching results over the next couple of days.

@Cstone. Great, how difficult do you think this is to switch the approach?

Note to the Phab: Adyen confirms that the setting to retry is in place now to retry 3DS declines WITHOUT 3DS to see if we can get an approval on the transaction. 5:05pm 28 July. Evelyn&team to monitor @RKumar_WMF Please note whether we see improvement on trxs after this time. Things will still decline but do we see them later approved?

@Adyen is changing the settings on our account (on 28 July), to retry declined 3DS transactions without 3DS to see if we get a lift in approval rates on Carte Bancaire. We will monitor this for a day or two and assess if approval rates improve.

From Adyen 28 July: //Hi Evelyn,

@spatton Sam, Adyen is saying our declines are legitimate 3DS declines and are wanting to change some settings on our account to see if that improves the situation. I'm learning about the implications of those changes now. Adyen is not reporting further bugs on our end so at the present time there doesn't seem to be anything FRTech can do here. I'll understand these settings, get them implemented and we can watch the results. I think that's our best option for now.

From Adyen thread FRTech is now looped in on: The reason your shoppers are seeing 'recurring' on their one-time payments is because you are specifying in your payment requests to use that you want to tokenise the card (ie. save the details for later usage). Here you are specifying the shopperReference, storePaymentMethod and recurringProcessingModel, which are values that are only needed to tokenise the card. If you don't want to save the details, you should omit these parameters. It's important to distinguish the process for doing this where your shoppers do and do not want to make recurring payments, only for one-off.

@Ejegg Adyen is assigning someone to talk to about this. I'll reach out when I have that person and run your questions by them and loop you in. Good that there are no actual recurring donations. @MBeat33 Is the list above the extent of the issue so far? It would be good to get order of magnitude if you have it.

@Cstone I received a one-time TY email as well.

@Ejegg. Does this indicate a problem on the Adyen end, i.e., they are not discarding the token as we are or is there something on our end we are invoking that we shouldn't be? Please let me know if I should escalate with Adyen. I've pinged them for input but will escalate it as necessary.

@Ejegg. I ran a transaction in Spain and Austria just now as one time and they showed up in the Adyen console as recurring. Both transactions had the post payment Monthly Convert prompt. NX8FF6P9QTLQ4V42, M94NFR6538FVMM42. I do not yet see the transactions in Civi. This may prove your theory. In the AT transaction, I saw a flash of the MC prompt but it immediately settled without allowing me time to consider the MC option - it appeared for only a second or less.

@mbeattie. We'll have Adyen check this out on their end. Thanks for flagging.

Noting this here for posterity.
Ok so why didn't the parent transaction make it to Civi in the example Rakhi posted (not pulled from acoustic or was it?)
10:31 AM <greg-g> greg
Based on the logs (from elliot's comment at https://phabricator.wikimedia.org/T313826#8105569): because it was using an incomplete/broken url. If the url has "%%isolang%%" in it things will fail. No real user should be presented with such an url.

@Ejegg Also, if we refund this transaction VQL7RPCTHS7WTM42, will the refund also not be reflected in Civi?

@Ejegg Hi Elliott, we are confused as to why the donations we ran yesterday are not in Civi. Are you saying we shouldn't copy links? Is this being worked on? Or is this the state of thing from here on?

@jgleeson @Ejegg Great news both! We will monitor France activity and ensure we are good to go with the campaign. Your attention here is much appreciated!! @RKumar_WMF fyi

@RKumar_WMF Hi Rakhi, Please follow this task concerning this issue. https://phabricator.wikimedia.org/T313092. A known problem is being worked on.

@RKumar_WMF Thank you, Rakhi, for overseeing that. Looks like we are good to resume in NL!

@greg. We need the call with Ingenico to understand their proposition on this, then we can assess in next sprint.

@RKumar_WMF Rakhi is working on a local (NL) user test and will report here.

@jgleeson Thanks Jack, I have added FRTech to the thread with Adyen concerning this matter and included your findings. Much appreciated.

Hi All, we are roughly at the same decline % since turning on 3DS. I have also set rules in the console to invoke 3DS which has been confirmed as valid by Adyen. I am inquiring further with Adyen as to why we are seeing such a high decline rate on Carte Bancaire and will advise further if they have any advice.

@jgleeson Thanks for the after hours response, Jack. Since we have to enable 3DS on everything vs nothing, I think everything should get a better result with it on. It's hard to say without testing so I say we proceed with the campaign with it on and see how we fare. It should hopefully help Carte Bancaire which has the biggest share in this market. @nisrael I think we are in the best place to proceed with email 2 now.

I set rules in the Adyen console to try to address this by device type. Monitoring impact of these settings to see if they apply.

@fr-tech. Adyen looked into this and advise the following: I have checked on our end, and everything looks good. And indeed you were able to process transactions until yesterday. I just attempted a test transaction, and it looks like the instruction email was never sent out from Wikimedia. The offer was promoted from our side no problem (and is visible in Customer Area), but there is no email sent to complete the payment.

@Pcoombe Hi Peter, yes, they all seem to be recurring/monthly. So what does it tell us that the form goes straight to MC? That seems to be our issue, correct?

I am able to see Ideal configured correctly on our account with Adyen with recent successful transactions. Does this point to a internal error?

@AMJohnson Hi Amber, I do not see any transactions for these individuals in Adyen as expected. I am confirming our setup is intact with Adyen.

yay!

@XenoRyet Thanks for the explanation, but if we have turned off MC presently by shutting down the campaign, how can we test this as being resolved?

To reiterate Peter's findings of earlier today, the error occurs when a donor selects give monthly (not give once). Those are the transactions that seem to intermittently reject to the donor but then they go on to clear in Adyen. I'm still finding this to be a problem. No resolution yet to recurring as I see it. Yes, tested in incognito mode, yes tested with various variants. Same issue.

@Cstone Hi Christine, I was able to get one payment through successfully with a TY page. However I tried again and got this error again:

yes, all of my testing is in incognito mode and without the variant as well.

Note that Thea is getting errors too, but I see her transactions getting through in Adyen. She THINKs they didn't but they did. Same problem as our users.

I tried from donate wiki as well with the same problem

I am using incognito when getting these errors fyi

I still see the authorization make it Adyen and the transaction is preparing to settle despite the error too. Same problem.... @XenoRyet @HNordeenWMF

I still am getting errors @HNordeenWMF telling me to contact donor relations. Here it is in German as I used Peter's link and even an alternate and same problem:

Thanks @XenoRyet. Good to know. I will have Adyen stand down on this. Note to the task: Sandra sent an email to Asjah to have her and I stop refunding as DR wishes to handle this.

@Fr-Tech. I just forwarded a thread from Adyen with a question. Can you respond there? They found an error int he log Peter sent: For that checkout page I was able to see an error in our logs:

Thank you. I am in the process of refunding the duplicates with the payments team.

Update from Adyen is as follows, which I have pushed back on as even if WMF is initiating two separate transactions, why would the user be getting an 'expired page' error. @fr-tech. Can we validate the flow all the same to be sure this isn't on our end. From Adyen: Given that these attempts are coming from webservice user ws@Company.WikimediaFoundation our recommendation would be to review the checkout flow in order to ensure that the integration is not triggering multiple attempts for the same checkout flow. See below one of the examples provided we query the shopperEmail and see two attempts with their own merchant reference and timestamp.

Escalated to Adyen executives having not heard from Adyen support as of 1:45 Pm Tuesday.

Noted. Likely why I couldn't reproduce. Mine were all one time with MC after.

I have sent this info over to Adyen for review.

So far, I have been unable to reproduce this in incognito testing and with several attempts in each bundle market.

I have initiated an inquiry with Adyen to see if they can determine the extent of the duplicates. Because they are the same email but two separate transactions within a few minutes, I can't get at the duplicate impact from their console. I'm also testing to see if I can duplicate the user experience that would cause this.

@Ejegg Sure, I was more thinking that the capability is built and tested so why not turn it on for all. We wouldn't go live with a campaign until the language is correct but the capability has been tested and proven so why not avoid about 14 additional requests into FRTech to avoid the noise for the other bundle markets.

Can we turn it on for all markets please since it passed the France/NL test?

Ok, I'm finishing a call. I'll IM you as soon as I'm off and I'll walk you
through it. Really odd the images didn't show up.

Hi Olga, did you see the images I posted in my last email?

Confirming there were recurring transactions which appeared in the Wednesday pretest: {F35249316}

Olga, so sorry, but I could not find the place to make the task 'visible
to public'. Pasting the screen shots below. Please let me know if this
suffices. I would be happy to run this through live with you on my laptop
if helpful:

@OTichonova Hi Olga, I am pasting a screen shot for your reference in how Apple Pay has been applied in the banner program:

@spatton Hi Sam, I think these questions are best answered by you concerning how we are positioning Apple Pay in the banner. Let me know if you don't agree.

Do we have to have it? Can it be removed? @AnnWF

@AnnWF Any update on this? Donor Relations is fielding several inquires around this @SHust

@FR-tech. Is this the user experience we need to accept for Gpay?

@fr-tech. Can we get an update on this? Was it addressed in the sprint above? We are having difficulty working fraud without this. Thanks

@AnnWF Hi Wenjun. I have entered this task which I believe is related to the problem we are seeing here. https://phabricator.wikimedia.org/T310548

We should still not be initiating a TY if the donation is stopped for review. @AnnWF Is this a bug?

We cannot work the anti-fraud queue when processing through Dlocal. There is no way to settle the transaction in their console. These transactions will sit there and never get settled as a result.

I do not see completed transactions in Dlocal or Civi for these donors. They aren't getting through the payment cycle. @Pcoombe for interest. Why are they getting a TY page?

Super! Thanks Elliott!

@Cstone Data is still taking a long time to download. The download is usually immediate but now it is taking 30mins to 1 hr if it downloads at all. We also are seeing 'downloading % of transactions' so we are wondering if we are getting all of the necessary data.