Change #1260779 had a related patch set uploaded (by Eileen; author: Eileen):

[wikimedia/fundraising/SmashPig@master] Bubble up auth_id & capture_id from Paypal

https://gerrit.wikimedia.org/r/1260779

Change #1260798 had a related patch set uploaded (by Ejegg; author: Ejegg):

[wikimedia/fundraising/SmashPig@master] Save PayPal 'auth id' to backend processor txn id

https://gerrit.wikimedia.org/r/1260798

Change #1260779 merged by jenkins-bot:

[wikimedia/fundraising/SmashPig@master] Bubble up auth_id & capture_id from Paypal

https://gerrit.wikimedia.org/r/1260779

Change #1260798 merged by jenkins-bot:

[wikimedia/fundraising/SmashPig@master] Save PayPal 'auth id' to backend processor txn id

https://gerrit.wikimedia.org/r/1260798

Change #1260822 had a related patch set uploaded (by Eileen; author: Eileen):

[wikimedia/fundraising/SmashPig@master] Put back backend_processor_txn_id

https://gerrit.wikimedia.org/r/1260822

Change #1260864 had a related patch set uploaded (by Eileen; author: Eileen):

[wikimedia/fundraising/crm@master] Update backend_processor_txn_id repair for bank transfer

https://gerrit.wikimedia.org/r/1260864

Change #1262313 had a related patch set uploaded (by Ejegg; author: Ejegg):

[mediawiki/extensions/DonationInterface@master] Stop sending session IDs as backend IDs (Adyen)

https://gerrit.wikimedia.org/r/1262313

Change #1262313 merged by jenkins-bot:

[mediawiki/extensions/DonationInterface@master] Stop sending session IDs as backend IDs (Adyen)

https://gerrit.wikimedia.org/r/1262313

Change #1262347 had a related patch set uploaded (by Eileen; author: Eileen):

[wikimedia/fundraising/crm@master] Extend repair to paypal -

https://gerrit.wikimedia.org/r/1262347

Change #1260864 merged by Eileen:

[wikimedia/fundraising/crm@master] Update backend_processor_txn_id repair for bank transfer

https://gerrit.wikimedia.org/r/1260864

Change #1262347 merged by Eileen:

[wikimedia/fundraising/crm@master] Extend repair to paypal -

https://gerrit.wikimedia.org/r/1262347

@Ejegg another case where we have a mismatch - perhaps should be a new phab

https://civicrm.wikimedia.org/civicrm/contact/view/contribution?reset=1&id=140812074&cid=63658630&action=view&context=contribution&selectedChild=contribute

In Civi
Backend Processor Transaction ID JCZST3WGRKXXWFB9
"payment_orchestrator_reconciliation_id: 4GRZsAXi4s6KNlaZJLtqCb **** - this actually maps to 8c247079-5432-4935-8c3d-36300dcccdd5
gravy gateway : 4c7df356-ae19-462c-a909-12eb6d9b1ca5

In Report
"backend_processor_txn_id":"BCT3S3BGCSD73735",
"payment_orchestrator_reconciliation_id":"2KKyXJfVrgQiQuagHYH9AT",
"gateway_txn_id":"4c7df356ae19-462ca909-12eb6d9b1ca5",

From Settlement

 {
"gateway":"gravy",
"gateway_txn_id":"4c7df356ae19-462ca909-12eb6d9b1ca5",
"backend_processor":"adyen",
"backend_txn_id":"BCT3S3BGCSD73735",
"audit_file_gateway":"adyen"
,"gateway_account":"WikimediaDonations",
"invoice_id":"245669506.1",
"settlement_batch_reference":"1156",
"exchange_rate":"1.000000000000000",
"settled_date":1773976767,
"contribution_tracking_id":245669506,
"email":"xxx",
"payment_method":"cc",
"payment_submethod":"mc",
"date":1773947616,
"backend_processor_txn_id":"BCT3S3BGCSD73735",
"payment_orchestrator_reconciliation_id":"2KKyXJfVrgQiQuagHYH9AT",
"type":"donation","modification_reference":"WTXCVWBHQM7C79B9",
"capture_id":"WTXCVWBHQM7C79B9",
"auth_id":"BCT3S3BGCSD73735",
"original_currency":"EUR",
"currency":"EUR",
"settled_currency":"EUR",
"gross":"10.40",
"original_total_amount":10.4,
"settled_fee_amount":"-0.47",
"fee":"0.47",
"original_fee_amount":-0.47,
"original_net_amount":"9.93",
"settled_gross":"9.93",
"settled_net_amount":"9.93",
"settled_total_amount":"10.40"}

There are two donations within 1 second of each other if you search on external_identifier in the gravy console.

We can still look into incrementing the sequence number before calling createPayment (instead of after) to help ensure unique order_ids even when two requests come in near-simultaneously.
Also I think opening up the IPN job runner's pending txn search to include resolved=1 would have let the duplicate in here, with the second one in getting a |dup-XXX suffix on its transaction ID.

Also, there are two rows in the pending table, each internally consistent. The job runner's search should also check by gateway txn ID when that's present, to make sure it gets the right one.

Change #1264664 had a related patch set uploaded (by Ejegg; author: Ejegg):

[wikimedia/fundraising/SmashPig@master] Pending message lookup defaults to including resolved

https://gerrit.wikimedia.org/r/1264664

Change #1264664 merged by jenkins-bot:

[wikimedia/fundraising/SmashPig@master] Pending message lookup defaults to including resolved

https://gerrit.wikimedia.org/r/1264664

Change #1264678 had a related patch set uploaded (by Jgleeson; author: Ejegg):

[wikimedia/fundraising/SmashPig@master] Add gateway txn hints to pending message lookups

https://gerrit.wikimedia.org/r/1264678

Change #1264678 merged by jenkins-bot:

[wikimedia/fundraising/SmashPig@master] Add gateway txn hints to pending message lookups

https://gerrit.wikimedia.org/r/1264678

Change #1260822 merged by jenkins-bot:

[wikimedia/fundraising/SmashPig@master] Put back backend_processor_txn_id

https://gerrit.wikimedia.org/r/1260822

@Ejegg - this is from today's adyen audit parse....

2026-04-22 02:52:02,137 ERROR civicrm.wmf.NOTICE: Starting process: get missing on /var/spool/auditadyen/incomingsettlement_d
etail_report_batch_1166.csv
2026-04-22 02:52:15,591 ERROR civicrm.wmf.INFO: Updating backend processor for XPN7QZGJTSHKXB69 {"auth_id":"XPN7QZGJTSHKXB69","cap
ture_id":"FW5SS2MW2G657X59","backend_processor":"adyen"}
2026-04-22 02:52:16,864 ERROR civicrm.wmf.INFO: Updating backend processor for RDKZMH4ZX3MRJW69 {"auth_id":"RDKZMH4ZX3MRJW69","cap
ture_id":"V8DXMTZJSVQW7669","backend_processor":"adyen"}
2026-04-22 02:52:17,732 ERROR civicrm.wmf.INFO: Updating backend processor for CG9P847BXRZK3S99 {"auth_id":"CG9P847BXRZK3S99","cap
ture_id":"VSFG872JFCT57X59","backend_processor":"adyen"}
2026-04-22 02:52:17,750 ERROR civicrm.wmf.INFO: Updating backend processor for D44FCRMKBKKJJB69 {"auth_id":"D44FCRMKBKKJJB69","cap
ture_id":"D6PBQPWGJM3PNV69","backend_processor":"adyen"}
2026-04-22 02:52:17,766 ERROR civicrm.wmf.INFO: Updating backend processor for VFSZGP6R2TGSRC35 {"auth_id":"VFSZGP6R2TGSRC35","cap
ture_id":"NZT2HHS9ZFTTN435","backend_processor":"adyen"}
2026-04-22 02:52:17,797 ERROR civicrm.wmf.INFO: Updating backend processor for VF4284466VML2V69 {"auth_id":"VF4284466VML2V69","cap
ture_id":"P45KR88TR48JDT69","backend_processor":"adyen"}
2026-04-22 02:52:17,960 ERROR civicrm.wmf.INFO: Updating backend processor for HXS2WGB68Q2Q5T99 {"auth_id":"HXS2WGB68Q2Q5T99","cap
ture_id":"PDD8KT8TR48JDT69","backend_processor":"adyen"}
2026-04-22 02:52:17,977 ERROR civicrm.wmf.INFO: Updating backend processor for SK449GZ223Z4F635 {"auth_id":"SK449GZ223Z4F635","cap
ture_id":"FKSVSL5QM9K2R535","backend_processor":"adyen"}
2026-04-22 02:52:18,033 ERROR civicrm.wmf.INFO: Updating backend processor for BQVD7NNKBKKJJB69 {"auth_id":"BQVD7NNKBKKJJB69","cap
ture_id":"Z8W49PFCVWGL5CB9","backend_processor":"adyen"}
2026-04-22 02:52:18,077 ERROR civicrm.wmf.INFO: Updating backend processor for K58Z7DLP2MJDG969 {"auth_id":"K58Z7DLP2MJDG969","cap
ture_id":"QDNFVZLQX82R8D69","backend_processor":"adyen"}
2026-04-22 02:52:18,106 ERROR civicrm.wmf.INFO: Updating backend processor for Z24CRFLP2MJDG969 {"auth_id":"Z24CRFLP2MJDG969","cap
ture_id":"N2RHF3MQX82R8D69","backend_processor":"adyen"}
2026-04-22 02:52:18,240 ERROR civicrm.wmf.INFO: Updating backend processor for QCV6HDQP5FMFT9B9 {"auth_id":"QCV6HDQP5FMFT9B9","cap
ture_id":"PJWMZFC57SFX7669","backend_processor":"adyen"}
2026-04-22 02:52:18,310 ERROR civicrm.wmf.INFO: Updating backend processor for TFPS6VFCFQ5TRC35 {"auth_id":"TFPS6VFCFQ5TRC35","cap
ture_id":"GDFK3C9TR48JDT69","backend_processor":"adyen"}

Aha, at least the first few of those are recurring charges. Thanks! I'll fix the backend IDs for that flow.

Change #1276724 had a related patch set uploaded (by Ejegg; author: Ejegg):

[wikimedia/fundraising/crm@master] Fix backend IDs sent to queue for recurring

https://gerrit.wikimedia.org/r/1276724

Original: I've just deployed the fix to send the right IDs to the queue for Gravy/Adyen recurring charges

Oops, no, that fix still needs review: https://gerrit.wikimedia.org/r/c/wikimedia/fundraising/crm/+/1276724

Change #1277183 had a related patch set uploaded (by Ejegg; author: Ejegg):

[wikimedia/fundraising/SmashPig@master] Gravy/Adyen: backend txn ID is auth ID

https://gerrit.wikimedia.org/r/1277183

Change #1277183 merged by jenkins-bot:

[wikimedia/fundraising/SmashPig@master] Gravy/Adyen: backend txn ID is auth ID

https://gerrit.wikimedia.org/r/1277183

Change #1279443 had a related patch set uploaded (by Ejegg; author: Ejegg):

[wikimedia/fundraising/crm@master] Always add backend IDs from recurring charges

https://gerrit.wikimedia.org/r/1279443

Change #1276724 merged by Eileen:

[wikimedia/fundraising/crm@master] Fix backend IDs sent to queue for recurring

https://gerrit.wikimedia.org/r/1276724

Change #1279443 merged by jenkins-bot:

[wikimedia/fundraising/crm@master] Always add backend IDs from recurring charges

https://gerrit.wikimedia.org/r/1279443