Page MenuHomePhabricator

Engage <> Gr4vy for Direct Mail CC and ACH (moto)
Open, Needs TriagePublic

Description

Gr4vy is working on a solution for us to process Direct Mail Credit Card gifts that come in to Engage through Adyen. Down the road we will also add ACH capabilities. We want to make sure that these gifts are coded in a way that differentiates them from other online gifts coming through Adyen. We have come up with a possible solution to add a "sub channel" = Direct Mail Upload which would be the channel for any CC or ACH Direct Mail gifts processed through Gr4vy/adyen. Finance will create a new GL code to assist in their recon of these gifts.

Gr4vy should be done with their work by end of February and then we can start testing. At that point we will coordinate with fr-tech to translate the IPN Gr4vy sends and make sure these gifts are showing in Civi correctly.

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Change #1243299 merged by jenkins-bot:

[wikimedia/fundraising/crm@master] Add direct mail upload channel, copy known GL codes to description

https://gerrit.wikimedia.org/r/1243299

Thank you! I will follow up again once we are ready to go live with this new coding/processing through Gr4vy.

Hi all, due to some extra data security work that is needed on Gr4vy's side the testing stage has been pushed back to the week of March 9.

AKanji-WMF subscribed.

As work is active on this I'm bringing this into Sprint.

MOTO has been enabled by Adyen and Gr4vy is doing some prep work before we are able to start live testing.

Noting that we are currently testing and working through some troubleshooting with Gr4vy and Engage. Looping in @MRitch-WMF for visibility.

Right now the transactions are in the adyen audit file but the adyen audit processor doesn't know what to do with them. We just need to update the audit code to handle these:

What's in the audit file:
payments_accounting_report_2026_04_29.csv:WikimediaFoundation,WikimediaDonations,DWPB9337Z8VVFV99,4yZ2n5y73flpvCHpBFlJtm,mc,mcsuperpremiumcredit,2026-04-29 13:17:09,PDT,Settled,NVG2DZN7TJZHTK69,2026-04-29 07:22:25,1172,,USD,24.25,USD,,1.000000000000000,0.00,-25.00,USD,24.25,,0.06,0.09,0.60,,0.0,US,06794P,12302026119001953422095,4yZ2n5y73flpvCHpBFlJtm,25.00,,,,,,,,,,,,,50016QLHGHMVL5PGM51WHEMF4MYT2,,,,

What the audit processor errors on:
civicrm.wmf.INFO: contribution not found using contribution_extra.gateway gravy and gateway_txn_id a386203e-656e-4b6a-a4b2-d2f73052fc26
2026-04-30 20:24:07,395 ERROR {"gateway":"gravy","gateway_txn_id":"a386203e-656e-4b6a-a4b2-d2f73052fc26","backend_processor":"adyen","backend_txn_id":"DWPB9337Z8VVFV99","audi
t_file_gateway":"adyen","gateway_account":"WikimediaDonations","invoice_id":"4yZ2n5y73flpvCHpBFlJtm","settlement_batch_reference":"1172","exchange_rate":"1.000000000000000","set
tled_date":1777493829,"contribution_tracking_id":null,"payment_method":"cc","payment_submethod":"mc","date":1777493829,"backend_processor_txn_id":"DWPB9337Z8VVFV99","payment_orc
hestrator_reconciliation_id":"4yZ2n5y73flpvCHpBFlJtm","type":"donation","modification_reference":"NVG2DZN7TJZHTK69","capture_id":"NVG2DZN7TJZHTK69","auth_id":"DWPB9337Z8VVFV99",
"original_currency":"USD","currency":"USD","gross":"25.00","original_total_amount":"25.00","fee":"0.75","original_fee_amount":"0.75","settled_gross":"24.25","settled_net_amount"
:"24.25","settled_currency":"USD","settled_fee_amount":"-0.75","original_net_amount":"24.25","settled_total_amount":"25.00"}

@MDemosWMF so my understanding is that we have 3 test transactions from Gravy so far & @Cstone & I have looked at them & Gravy needs to update both the IPN and what is in the payments & settlement file (ie the Metadata field) to provide more information - including

  • contact information
    • first_name
    • last_name
    • email -contact_id
  • package
  • appeal
  • channel
  • engage_batch
  • engage unique reference
  • some field like 'payment_source' that would be === 'engage'
  • there is a field is_recurring - in the IPN - not sure if that is relevant

GRAVY 385d9717-8b4f-4d33-88f0-78bf304f87f1 1iML9ORlPjfsEcxzstITRJ
GRAVY 31034658-7762-4b2e-b84c-d11a5f6003e1

@Eileenmcnaughton Are Fund and Gift Type already set up? I know those are in the metadata as well.

@MDemosWMF hmm no - I think I just thought they were always the same -but better to send them in!

Change #1281675 had a related patch set uploaded (by Eileen; author: Eileen):

[wikimedia/fundraising/crm@master] Reduce audit files boilerplate

https://gerrit.wikimedia.org/r/1281675

Noting here we have successfully processed 3 transactions through the Gr4vy flow and metadata looks good. However we are not seeing the donations in Civi. @Eileenmcnaughton is investigating.

Once this is resolved next step is to test with recurring CC processing. This will unlock further capability for DM offerings.

Going back and forth with gravy in slack. Trying to get donor info in the meta data field so we can import them easier

Note I'm gonna manually create the attempted ones in the meantime

OK - I'm fixing the backend_gateway, gateway, gateway, gateway_txn_id, backend_gateway_id on this one manually so we can clear it out - we need to be more co-ordinated with our next tests because these are still very much breaking things for us when they happen so we should be planning the timing together

https://civicrm.wikimedia.org/civicrm/contact/view/contribution?reset=1&id=144852880&cid=66057760&action=view&context=contribution&selectedChild=contribute

144852880

Fix for 144852880

echo '{"values":{"contribution_extra.gateway":"gravy","contribution_extra.backend_processor":"adyen","contribution_extra.gateway_txn_id":"0fddd661-5aa1-40c3-bf98-1d6e0445b771","contribution_extra.backend_processor_txn_id":"XTS8TNPQNM6MVLB9","contribution_extra.auth_id":"XTS8TNPQNM6MVLB9","contribution_extra.capture_id":"MD9VRN4Q8VLTZC35","contribution_extra.payment_orchestrator_reconciliation_id":"TwGQEoV8slYeKe8CtLCb3"},"where":[["id","=",144852880]]}' | wmf-cv api4 Contribution.update --in=json

OK - the process for fixing these is pretty painful but

the next one is https://civicrm.wikimedia.org/civicrm/contact/view?reset=1&cid=68545573

id 144852775

Note - that the gravy stuff is putting the contact ID in our order_id field - ie "orchestrator_tx_ref"":""68545573" is the contact_id - that is the field where our contribution_tracking ID goes

  • still getting info on this one

echo '{"values":{"contribution_extra.gateway":"gravy","contribution_extra.backend_processor":"adyen","contribution_extra.gateway_txn_id":"d35b414b-7e67-4fc0-9757-d7fbc4c43b2d","contribution_extra.backend_processor_txn_id":"NJT3S92J2XVXF669","contribution_extra.auth_id":"NJT3S92J2XVXF669","contribution_extra.capture_id":"JGZ62LPNP34H6T69","contribution_extra.payment_orchestrator_reconciliation_id":"6Qp5FTmxgkGnYIOuqsZAoX"},"where":[["id","=",144852775]]}' | wmf-cv api4 Contribution.update --in=json

Found the 3rd one

echo '{"values":{"contribution_extra.gateway":"gravy","contribution_extra.backend_processor":"adyen","contribution_extra.gateway_txn_id":"e8a27515-05a6-4429-adfa-6f1cb7865372","contribution_extra.backend_processor_txn_id":"GVKK9P6RJFBL2V69","contribution_extra.auth_id":"GVKK9P6RJFBL2V69","contribution_extra.capture_id":"VQHJTQH8P43X8PB9","contribution_extra.payment_orchestrator_reconciliation_id":"74ySi4WMCXDCMaGaSmIRXO"},"where":[["id","="144852767"]]}' | wmf-cv api4 Contribution.update --in=json

{"gateway":"gravy","gateway_txn_id":"e8a27515-05a6-4429-adfa-6f1cb7865372","backend_processor":"adyen","backend_txn_id":"GVKK9P6RJFBL2V69","audit_file_gateway":"adyen","gateway_account":"WikimediaDonations","invoice_id":"32427985","settlement_batch_reference":"1218","exchange_rate":"1.000000000000000","settled_date":1781623183,"contribution_tracking_id":32427985,"payment_method":"cc","payment_submethod":"visa","date":1781623183,"backend_processor_txn_id":"GVKK9P6RJFBL2V69","payment_orchestrator_reconciliation_id":"74ySi4WMCXDCMaGaSmIRXO","type":"donation","modification_reference":"VQHJTQH8P43X8PB9","capture_id":"VQHJTQH8P43X8PB9","auth_id":"GVKK9P6RJFBL2V69","original_currency":"USD","currency":"USD","gross":"25.00","original_total_amount":"25.00","fee":"0.52","original_fee_amount":"0.52","settled_gross":"24.48","settled_net_amount":"24.48","settled_currency":"USD","settled_fee_amount":"-0.52","original_net_amount":"24.48","settled_total_amount":"25.00"}

@Eileenmcnaughton - @Cstone Said in slack that these gifts had actually been imported in Civi from an Engage file. Just want to make sure we are not duplicating anything. Are you updating those imported gifts with the Gr4vy info?

@MDemosWMF yes - I found & updated them - it was pretty painful but those 3 are not causing problems anymore

Gr4vy is currently working on adding the customer details to the webhook. No ETA for that yet.

Action Items from 7/14 call:

-WMF to evaluate the best approach to ingest the webhooks + query for details
-Gr4vy to update metadata from cid to customer_id
-Gr4vy to explore returning additional_identifiers in webhook so the auth id is available in the capture webhook

@Cstone please add any further info here

@AKanji-WMF Would you be able to assist with asessing how this work will fit into fr-tech's other priorities? A timeline for this will help @MRitch-WMF plan when we can incorporate recurring CC into future DM campaigns. Thanks!

Change #1311894 had a related patch set uploaded (by Cstone; author: Cstone):

[wikimedia/fundraising/SmashPig@master] WIP: Handle Moto webhooks

https://gerrit.wikimedia.org/r/1311894

Change #1311895 had a related patch set uploaded (by Cstone; author: Cstone):

[wikimedia/fundraising/SmashPig@master] WIP: Handle Moto webhooks

https://gerrit.wikimedia.org/r/1311895

Change #1311894 abandoned by Cstone:

[wikimedia/fundraising/SmashPig@master] WIP: Handle Moto webhooks

https://gerrit.wikimedia.org/r/1311894

Gr4vy to update metadata from cid to customer_id > This will be week of 7/20.

On the webhook changes, this is pending the Gr4vy team to evaluate. There is a potential workaround but they need @Cstone to review.

"I was checking out the additional_identifiers and noticed that we have the additional_identifiers in the transaction query response. Is this potentially something you can use until we return the data in the webhook?

The payment_service_authorization_id is the Adyen reference for the authorization_id that you are looking for I believe?

    "additional_identifiers": {
        "payment_service_authorization_id": "TK8LPLFN6FXH9275",
        "payment_service_capture_id": "F2R6MCWRWDNZW575",
        "payment_service_processor_id": null
    },

@Cstone Thinks the above is doable, but will double check.

Cstone renamed this task from Engage <> Gr4vy for Direct Mail CC and ACH to Engage <> Gr4vy for Direct Mail CC and ACH (moto).Jul 21 2026, 3:48 PM

Gr4vy has said metadata change should be available next week, no update on webhook work yet.

Gr4vy metadata updates are now in sandbox and production. @Cstone

Update on ACH MOTO. We should be able to use Adyen's ACH services. Gr4vy has said they are approximately 90% ready with the implementation. They currently need to add a connection option to configure the secCode, as it cannot be determined solely by the MOTO flag due to MOTO capturing both phone and mail use cases without differentiation. They have said this should be ready week of 8/17.

@Cstone wondering if it's better to separate the ACH work into a separate phab?

@MDemosWMF i think the way the current phabs are its not tied to a specific payment method but to the way we get the donation so the current phabs should be able to cover both.

Gr4vy ACH update: They've made the changes to add the secCode for ACH. It runs OK on our account and we are getting a successful response.
Charlotte will run a test against the wiki sandbox so we can get Charlotte to check and ensure the transaction data is correct.

Change #1311895 abandoned by Cstone:

[wikimedia/fundraising/SmashPig@master] WIP: Handle Moto webhooks

https://gerrit.wikimedia.org/r/1311895

gravy has the test ACH credentials, waiting on them to finish their implementation now

@Cstone Any updates on the credit card Gr4vy work? I see a couple subtasks for the webhook and capture job in progress. Let me know if there is anything further we need from Gr4vy.

@dmiranda Just checking in on how things are looking with the credit card work on this task, thanks!