'Expected field payments_final_status bound for table payments_initial' failmails
Open, HighPublic
Actions

Assigned To

None

Authored By

	jgleeson
	Sep 5 2024, 4:20 PM

Description

We've had a bunch of these over last night and today. @Cstone tracked one down, and it was due to a Gravy refund bug but the handful I looked at were Adyen iDeal and SEPA payments so it feels like something fishy is happening

Example:
2024-09-05T11:29:02+00:00 [ERROR] Error processing message, moving to damaged store.
Civi\WMFException\FredgeDataValidationException@/srv/org.wikimedia.civicrm/drupal/sites/default/civicrm/extensions/wmf-civicrm/Civi/WMFQueue/PaymentsInitQueueConsumer.php:69 (FREDGE adyen-212991633.1: Expected field payments_final_status bound for table payments_initial not present! Dropping message on floor.
Source: array (
))

Details

Subject	Repo	Branch	Lines +/-
Send donor to fail page on Adyen fail	mediawiki/extensions/DonationInterface	master	+1 -3
Log error when sending bad params to createPayment	wikimedia/fundraising/SmashPig	master	+12 -0
Always set FinalStatus failed on no response status	wikimedia/fundraising/SmashPig	master	+4 -5

Customize query in gerrit

Related Objects

Mentioned Here: rMWed2d78b30ea4: Update git submodules

Event Timeline

jgleeson created this task.Sep 5 2024, 4:20 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSep 5 2024, 4:20 PM

AKanji-WMF moved this task from Triage to DRI Backlog on the Fundraising-Backlog board.Sep 5 2024, 5:03 PM

XenoRyet triaged this task as High priority.Sep 30 2024, 8:14 PM

Cstone claimed this task.Oct 1 2024, 2:49 AM

Cstone moved this task from Backlog to In Progress on the Fundraising Tech - Chaos Crew board.

This is interesting, something is happening that's causing a null response and then we lose the final status. Locally stepping through on a null response it's being caught and getting a final status of failed

Cstone removed Cstone as the assignee of this task.Tue, Oct 8, 3:44 AM

Cstone moved this task from In Progress to Backlog on the Fundraising Tech - Chaos Crew board.

Ejegg moved this task from Backlog to In Progress on the Fundraising Tech - Chaos Crew board.Thu, Oct 10, 5:41 PM

The latest instance of this happened even with Dami's fix deployed to SmashPig

https://civicrm.wikimedia.org/civicrm/damaged/edit?action=update&id=651456&reset=1 happened on 10/09 with payments-wiki at commit ed2d78b3 which includes SmashPig v0.8.14

Change #1079332 had a related patch set uploaded (by Ejegg; author: Ejegg):

[wikimedia/fundraising/SmashPig@master] Always set FinalStatus failed on no response status

https://gerrit.wikimedia.org/r/1079332

gerritbot added a project: Patch-For-Review.Thu, Oct 10, 5:49 PM

Ejegg moved this task from In Progress to Ready for Review on the Fundraising Tech - Chaos Crew board.Thu, Oct 10, 5:50 PM

Change #1079332 merged by jenkins-bot:

[wikimedia/fundraising/SmashPig@master] Always set FinalStatus failed on no response status

https://gerrit.wikimedia.org/r/1079332

More odd stuff looking at this failed ACH from very recently: https://civicrm.wikimedia.org/civicrm/damaged/edit?action=update&id=651557&reset=1

the PHP session ID is associated with 3 different contribution tracking IDs, so the donor definitely tried multiple times
first attempt was authorized but rejected for fraud
second attempt had the null stuff
no log line with any of the ct_ids is found with the 'Outbound data' or 'Initiating cURL' at the appropriate timestamp (these should be prefixed by the SmashPig logger) but we do see two 'Initiating cURL' lines at the appropriate timestamp with NO ct_id.

Maintenance_bot removed a project: Patch-For-Review.Thu, Oct 10, 6:30 PM

Steps to replicate locally:

try doing an ACH on Adyen with the correct test credentials (any name, any acct number, routing number 011000138 or 121000358) but force it to be fraud-rejected (I added $this->setValidationAction(ValidationAction::REJECT); in runFraudFiltersIfNeeded)
because FinalStatus is PENDING-POKE and ValidationAction is REJECT, we clear out the whole session, including the submethod.
red text appears above the form but form is not disabled
click the submit button again. payment submethod is NOT re-submitted and since it's not present in the session it doesn't get sent to the SmashPig createPayment call.
SmashPig logic depends on submethod to go down the right path. issuer_id, iban, and recurring_payment_token are also all blank, so none of the paths to actually make an Adyen API call are followed, and $rawResponse stays null: https://phabricator.wikimedia.org/diffusion/WFSP/browse/master/PaymentProviders/Adyen/BankTransferPaymentProvider.php$29

AnnWF moved this task from Ready for Review to Pending Deployment on the Fundraising Tech - Chaos Crew board.Thu, Oct 10, 8:58 PM

I think we need to do some more work on this. Specifically, we should send the donor to the error page when the submission fails rather then clearing out the session and letting them submit again. Moving back to 'In Progress'.

Change #1081173 had a related patch set uploaded (by Ejegg; author: Ejegg):

[mediawiki/extensions/DonationInterface@master] Send donor to fail page on Adyen fail

https://gerrit.wikimedia.org/r/1081173

gerritbot added a project: Patch-For-Review.Thu, Oct 17, 2:15 PM

Change #1081177 had a related patch set uploaded (by Ejegg; author: Ejegg):

[wikimedia/fundraising/SmashPig@master] Log error when sending bad params to createPayment

https://gerrit.wikimedia.org/r/1081177

OK, I think those two patches should catch the specific issue that was causing things on ACH payments.

AnnWF moved this task from Ready for Review to Pending Deployment on the Fundraising Tech - Chaos Crew board.Mon, Oct 21, 6:14 PM

Change #1081177 merged by jenkins-bot:

[wikimedia/fundraising/SmashPig@master] Log error when sending bad params to createPayment

https://gerrit.wikimedia.org/r/1081177

Change #1081173 merged by jenkins-bot: