These are all our gravy venmo test transactions linking T409599: Venmo issues from first round of testing

@Cstone is this a case where gravy has 'stolen' our contribution_tracking_id field?
settlement_batch_report_2025-11-20.json

{"contribution_tracking_id":"4dKvU4tsIv5DZRxK4jYbib",
"invoice_id":"4dKvU4tsIv5DZRxK4jYbib","payment_method":"venmo","currency
":"USD","email":null,"phone":null,
"first_name":"Ch**","last_name":null,"gross":"1.03","date":"2025-11-21T12:53:12.000000Z","gateway_txn_id"
:"dHJhbnNhY3Rpb25fZXAwbnN5d2Y"}

MariaDB [civicrm]> select * FROM civicrm_transaction_log WHERE message LIKE '%4dKvU4tsIv5DZRxK4jYbib%'\G
Empty set (59.709 sec)

MariaDB [civicrm]> select * FROM civicrm_transaction_log WHERE message LIKE '%dHJhbnNhY3Rpb25fZXAwbnN5d2Y%'\G
Empty set (58.366 sec)

MariaDB [civicrm]> select * FROM damaged_view WHERE message LIKE '%4dKvU4tsIv5DZRxK4jYbib%'\G
Empty set (0.728 sec)

MariaDB [civicrm]> select * FROM damaged_view WHERE message LIKE '%dHJhbnNhY3Rpb25fZXAwbnN5d2Y%'\G
Empty set (0.333 sec)

Just some more data points - basically the gravy ID in the braintree log has 0% cross over with the gravy ID in the transaction log & they don't give us our contribution_tracking_id back...

The invoice ID of 4dKvU4tsIv5DZRxK4jYbib maps to 984639ea-59f4-4ab0-b750-bf77f0592e21 - this is not in the transaction log entry which has

             id: 133982198
           date: 2025-10-21 18:29:31
        gateway: gravy
gateway_account: WikimediaDonations
       order_id: 238976819.1
 gateway_txn_id: 0206d242-9b7a-4476-9ae9-03a634890974
        message: {
            "gateway_txn_id":"0206d242-9b7a-4476-9ae9-03a634890974",
            "response":false,
            "gateway_account":"WikimediaDonations",
            "fee":0,
            "gross":"1.03",
            "backend_processor":"braintree",
            "contribution_tracking_id":"238976819",
            "country":"US",
            "currency":"USD",
            "email":"",
            "frequency_unit":"month",
            "gateway":"gravy",
            "language":"en",
            "order_id":"238976819.1",
            "payment_method":"venmo",
            "payment_orchestrator_reconciliation_id":"3p6K7pa7zfvgpz7XFYr6i",
            "payment_submethod":"",
            "recurring":"1",
            "recurring_payment_token":"abcd",
            "user_ip":"xx.xx.x.x,xx",
            "utm_campaign":"spontaneous",
            "utm_medium":"spontaneous",
            "utm_source":"fr-redir.default~default~default~default~control.rvenmo",
            "date":1761071371,
            "source_name":"DonationInterface",
            "source_type":"payments",
            "source_host":"payments1006",
            "source_run_id":4179683,
            "source_version":"bb9ad03a50a2f84b0cb5875486d130742bd21d45",
            "source_enqueued_time":1761071371
        }
    payment_method: venmo
    is_resolved: 1
*************************** 2. row ***************************
             id: 133982212
           date: 2025-10-21 18:29:46
        gateway: gravy
gateway_account: WikimediaDonations
       order_id: 238976819.1
 gateway_txn_id: 0206d242-9b7a-4476-9ae9-03a634890974
        message: {
          "gateway_txn_id":"0206d242-9b7a-4476-9ae9-03a634890974",
          "response":false,
          "gateway_account":"WikimediaDonations",
          "fee":0,
          "gross":"1.03",
          "backend_processor":"braintree",
          "backend_processor_txn_id":"dHJhbnNhY3Rpb25fYW5iMW54OXA",
          "contribution_tracking_id":"238976819",
          "country":"US",
          "currency":"USD",
          "email":"x@x.com",
          "first_name":"C",
            "frequency_unit":"month",
            "gateway":"gravy",
            "language":"en",
            "last_name":"S",
            "order_id":"238976819.1",
            "payment_method":"venmo",
            "payment_orchestrator_reconciliation_id":"3p6K7pa7zfvgpz7XFYr6i",
            "payment_submethod":"",
            "recurring":"1",
            "recurring_payment_token":"1234",
            "user_ip":"xx.xx.xx.xx",
            "utm_campaign":"spontaneous",
            "utm_medium":"spontaneous",
            "utm_source":"fr-redir.default~default~default~default~control.rvenmo",
            "date":1761071386,
            "external_identifier":"CR**",
            "source_name":"DonationInterface",
            "source_type":"payments",
            "source_host":"payments1006",
            "source_run_id":317904,
            "source_version":"bb9ad03a50a2f84b0cb5875486d130742bd21d45",
            "source_enqueued_time":1761071386
     }
 payment_method: venmo
    is_resolved: 1

for dHJhbnNhY3Rpb25fZXAwbnN5d2Y in braintree

Connector transaction ID
dHJhbnNhY3Rpb25fZXAwbnN5d2Y

Processor ID
1PXDY9

Legacy ID
ep0nsywf

External identifier
238976819.3

Reconciliation ID
4dKvU4tsIv5DZRxK4jYbib

Transaction ID
984639ea-59f4-4ab0-b750-bf77f0592e21

In civi:
Backend Processor braintree
Backend Processor Transaction ID dHJhbnNhY3Rpb25fZXAwbnN5d2Y
Payment Orchestrator Reconciliation ID 4dKvU4tsIv5DZRxK4jYbib
Gateway Transaction ID 984639ea-59f4-4ab0-b750-bf77f0592e21

Change #1211919 had a related patch set uploaded (by Eileen; author: Eileen):

[wikimedia/fundraising/SmashPig@master] Surface orchestrator_reconciliation_id where present

https://gerrit.wikimedia.org/r/1211919

Change #1211935 had a related patch set uploaded (by Eileen; author: Eileen):

[wikimedia/fundraising/crm@master] Ensure that gravy venmo are 'found'

https://gerrit.wikimedia.org/r/1211935

Change #1211919 merged by jenkins-bot:

[wikimedia/fundraising/SmashPig@master] Surface orchestrator_reconciliation_id where present

https://gerrit.wikimedia.org/r/1211919

so I rolled out that braintree patch & I think the one I was looking at cleared but I have a new one...

• not gravy

contribution_tracking_id 241008010
gateway_txn_id dHJhbnNhY3Rpb25fMTQ2MjlkNGQ

NO pending transaction record

• contact does not have a contribution for that date - https://civicrm.wikimedia.org/civicrm/contact/view?reset=1&cid=17182841

3 Dec

• lots of stuff like this Dec 3 17:26:45 payments1006 SmashPig-Braintree: braintree::241008010:241008010.1 | Preparing to send POST request to https://payments.braintree-api.com/graphql | |
• and this Dec 3 17:26:38 payments1006 braintree_gateway_fraud: 241008010:241008010.1 "utm" "{\"utm_campaign\":\"WMF_FR_C2526_en6C_dsk_1202\",\"utm_medium\":\"sitenotice\",\"utm_source\":\"B2526_120313_en6C_dsk_p1_lg_hdr_cnt.no-LP.venmo\",\"utm_key\":\"otherAmt_0~ptf_1~vw_1194~vh_970~time_58\"}"

Dec 3 17:26:38 payments1006 braintree_gateway_fraud: 241008010:241008010.1 Pushing transaction to payments-antifraud queue.

I scanned the logs for the error string that is returned after sending the request, since this are all direct connection Braintree, I used this grep command to search for the failures in the logs:

zgrep --only-matching "braintree::.*cURL transaction to https://payments.braintree-api.com/graphql failed: (6) Could not resolve host: payments.braintree-api.com." /srv/archive/frlog1002/logs/payments-misc-20251204.gz | sort | uniq -d | wc -l

The results showed that 33 Braintree transactions between 17:25 and 17:30 UTC failed due to probably a DNS outage. The PayPal status page does not show any reports of failures within that time bound. However this time bound does coincide with the DDoS attack on that same day, might be related as there as not being any recurrence since then and none prior.

I manually checked all 33 transactions and of the total, only 6 transactions actually went through. One of this transactions was stuck on Authorized, which I manually settled, waiting to be pushed into Civi.

The first two transactions might require manual importing into Civi.

Attached are all the failed ordered IDs.

Change #1211935 merged by jenkins-bot:

[wikimedia/fundraising/crm@master] Add test for fix to Ensure that gravy venmo are 'found'

https://gerrit.wikimedia.org/r/1211935

I used isMakeMissing to create the second of the 2 missing above

contribution_tracking_id 241008010
gateway_txn_id dHJhbnNhY3Rpb25fMTQ2MjlkNGQ

241008010.1 14629d4d Settled No

I think this is probably closable now - I used the audit makeMissing to fill in gaps & am trying to get to a clean slate going forwards....