Page MenuHomePhabricator

AMEX rejected transactions through Ingenico are to be ported to Adyen
Closed, ResolvedPublic

Description

FRTech, Ingenico/ Worldline have apprised us that , for Amex, the currencies below were removed from our configuration as they’re no longer supported. I would like to recover this small set of recurring donors by porting them to Adyen and resuming the recurring billing there. This is also a discovery opportunity to trial a migration before we seek to migrate all of the 165k recurring donors from Ingenico to Adyen which is on the roadmap and is a much more sensitive endeavor.

image.png (520Γ—929 px, 253 KB)

Here is the migration process from Adyen. As you will see there is an action on us to ensure we can bill to a different provider for these migrated tokens. Can we support that the recurring charge be performed at two different processors and what of the shopper reference how will we handle that?

I have sent the format to Ingenico for their reaction as to whether they can align with the format and will advise further here. In the meantime, if they can, can we support this migration?

From Adyen:
Before we begin the import process, you'll want to ensure the following steps have been taken:

Adjust your technical processes so tokenization and recurring charge processes can be performed at two different payment service providers (PSP).
Decide on an internal shopper reference to use as the Adyen shopperReference.

Adyen uses a reference to the shopper rather than the payment details for tokenization.
Contact your current PSP and check if you can use the internal shopper reference you have chosen. If not, choose one of their IDs to link with your Adyen shopperReference.
Check their preferred method of transferring files to Adyen.
If they can export the data in the required format, you can contact us to begin the process. We will create a custom PGP key which will speed up your card import process. For more information on required formats, see recurring payments details and referral lists.
Otherwise, ask your old provider to encrypt the file format with the Adyen PGP key so that Adyen can perform custom manipulation of the file format afterwards.

The files containing the data to be imported should be formatted as follows:

Create a file in CSV (Comma Separated Values) format as per RFC 4180, in accordance with the following requirements:
The first line contains the names of the fields. You do not have to use all field names and you do not have to use them in the same order as listed on this page.
Each subsequent line contains the fields for a single recurring contract. In other words: Each recurring contract is on a separate line.
If a field doesn't apply to the recurring contract that the line describes, specifying a value is skipped.
Field names and values are case-sensitive.
Multi-line fields are not allowed.
The maximum file length is 1,000,000 lines.
Save the file in UTF-8 format (to support non-western characters).
Encrypt the file using the PGP public key provided to you by either our Support Team, or your Adyen contact.
For more information, see our PGP encryption documentation.
Send the file to Adyen.

You can find an example input file, as well as a list of the input file fields, in our "Import payment details for recurring payments" documentation. For a general overview of the import process, you can look over our "Import data from another payment provider" documentation.

Event Timeline

@EMartin fr-tech will need to make some updates in the Civi database to synchronize with this migration. Specifically, we will need to update the payment processor from Ingenico to Adyen, and each donation's token from the old Ingenico token to a new Adyen token.

Since the file will include card numbers and expiry dates, I guess it will be have to be generated by Ingenico. We will want them to put an identifier we have in our DB for each subscription in the EchoData field as mentioned in Adyen's docs. Maybe the Ingenico recurring payment token would make sense here?

When Adyen has imported the file from them, fr-tech will need the output file to map from the old tokens to the new tokens. We will need to write a bit of code to do that mapping.

Thanks @Ejegg . Ingenico will transfer the file with the PII directly to Adyen. I will tell them they need to include the Ingenico recurring token in the EchoData field.

Email from Adyen:

Laura Fernandez (Adyen)

Aug 17, 2023, 09:27 GMT+2

Hi Evelyn,

We were able to connect to the SFTP and retrieve the following sample file: ensr4525_sample_send.csv.gpg. However, we were not able to decrypt it. We noticed that the file was encrypted using the following key: 3C6769B22E2AEE43.

We would require your help to encrypt the files using your Adyen merchant specific key CFDCF27F66837B4C attached for company account: WikimediaFoundation.

You can refer to our documentation below to validate the key.
https://docs.adyen.com/development-resources/migrating-payment-data/import-payment-details-for-recurring-payments/

Let me know if you have any follow up questions.

I heard back from Dylan and we will use the Order ID field in answer to Adyen's question. I have let them know.

Before procceding to import these tokens I'd like to confirm with Wikimedia team the following:

  1. The file is on a transaction level, this would mean that is possible we do not have the latest card data for the token. Ideally, to ensure usability of the tokens imported we would require to have an aggregated report with possible the most recent cards for the shoppers.
  1. We see the following new fields: SCHEMETRANSACTIONID (equivalent to the networkTXReference in Adyen), PROFILETOKEN, and Alternative merchantref. For shopperReference field would you still want to move forward with Alternative merchantref ?

@EMartin and @MBeat33 - There are only 6 tokens in the CSV attached to the email chain (each repeated 3x for some reason). All of them seem to be donors whose last donation was in June and who have received an email saying that their recurring donation has failed.

Are we OK just choosing one of them to reactivate, as long as they haven't started a new recurring donation?

Also, @EMartin is there any larger CSV available?

@Ejegg . The tokens for the deprecated currencies (above) were few in number. It is correct that they have not been billed since June as it took Ingenico months to get us the data after they had deprecated the currency. I am ok to choose one of them to reactivate. I see that 135527797.1 , 146810647.1 , 20136157.1. have not initiated a new payment with us since June. There is not another file, but ALL of the Ingenico volume will be ported by end of week (per Ingenico) so we'd like to prove out that these few work without disruption to the donors as you bill the tokens on Adyen.

Thanks @EMartin . I'll create a script to read the CSV and update all the Civi database entries to charge the donations via Adyen, then will attempt charging the one with the cycle day closest to now.

Change 962097 had a related patch set uploaded (by Ejegg; author: Ejegg):

[wikimedia/fundraising/crm@master] Migrate Ingenico tokens to Adyen

https://gerrit.wikimedia.org/r/962097

@Ejegg Thank you! I'll watch this task to ensure I review it in the console for the expected outcome. I am monitoring the following shopper ID's in the Adyen console to see their rebilling on Adyen: 146810647.1, 117201456.1, 135530208.1 since their card expiration dates are still valid.

Change 962097 merged by jenkins-bot:

[wikimedia/fundraising/crm@master] Migrate Ingenico tokens to Adyen

https://gerrit.wikimedia.org/r/962097

Change 962745 had a related patch set uploaded (by Ejegg; author: Ejegg):

[wikimedia/fundraising/crm@master] Deal with 1:many token migration

https://gerrit.wikimedia.org/r/962745

@EMartin some good news, we got one successful transaction charged on a migrated AMEX token:

https://ca-live.adyen.com/ca/ca/accounts/showTx.shtml?pspReference=VDXC7TQM9969BV62&txType=Payment
https://civicrm.wikimedia.org/civicrm/contact/view?reset=1&cid=47994811&selectedChild=contributions

I'm still working out the kinks in the migration script. Turns out there are some places where a single Ingenico token maps to multiple Adyen tokens (one for each subscription). Always some fun edge cases in our big old dataset!

@Ejegg I see the transaction in the Adyen console as well!

image.png (245Γ—1 px, 43 KB)

Change 963164 had a related patch set uploaded (by Ejegg; author: Ejegg):

[wikimedia/fundraising/crm@master] Actually execute the recur update

https://gerrit.wikimedia.org/r/963164

Change 963164 merged by jenkins-bot:

[wikimedia/fundraising/crm@master] Actually execute the recur update

https://gerrit.wikimedia.org/r/963164

Change 963424 had a related patch set uploaded (by Ejegg; author: Ejegg):

[wikimedia/fundraising/crm@master] Add batch and offset options to token migrator

https://gerrit.wikimedia.org/r/963424

Change 962745 merged by jenkins-bot:

[wikimedia/fundraising/crm@master] Deal with 1:many token migration

https://gerrit.wikimedia.org/r/962745

Change 963424 merged by jenkins-bot:

[wikimedia/fundraising/crm@master] Add batch and offset options to token migrator

https://gerrit.wikimedia.org/r/963424