Donations message with empty body!
Closed, ResolvedPublic2 Estimated Story Points
Actions

Description

globalcollect-1816361820 had the following headers but no body. Note the "redelivered" flag.

20:31:36              [redelivered] => true
20:31:36              [gateway] => globalcollect
20:31:36              [source_enqueued_time] => 1468613800
20:31:36              [message-id] =>
20:31:36  ID:silicon-54591-1463622100074-2:1676325:-1:1:1
20:31:36              [source_type] => payments
20:31:36              [source_host] => payments1003
20:31:36              [destination] => /queue/donations
20:31:36              [timestamp] => 1468613800058
20:31:36              [expires] => 0
20:31:36              [persistent] => true
20:31:36              [source_run_id] => 74718
20:31:36              [priority] => 4
20:31:36              [php-message-class] =>
20:31:36  SmashPig\CrmLink\Messages\DonationInterfaceMessage
20:31:36              [source_name] => DonationInterface
20:31:36              [source_version] =>
20:31:36  8bf6e911eb43a2d369bf656f07d1b51be0a54f6c
20:31:36              [correlation-id] => globalcollect-1816361820

Details

Subject	Repo	Branch	Lines +/-
Force bad encodings to approximate UTF-8	mediawiki/extensions/DonationInterface	deployment	+140 -52
Force bad encodings to approximate UTF-8	mediawiki/extensions/DonationInterface	master	+140 -52
[WIP] Reproduce Ingenico name encoding bug	mediawiki/extensions/DonationInterface	master	+64 -0
Log json_encode errors	mediawiki/extensions/DonationInterface	deployment	+5 -0
Log json_encode errors	mediawiki/extensions/DonationInterface	master	+5 -0
Log completed donations	mediawiki/extensions/DonationInterface	deployment	+23 -1
Log completed donations	mediawiki/extensions/DonationInterface	master	+263 -2
Fail if the message body is missing	wikimedia/fundraising/crm	master	+4 -0

Customize query in gerrit

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Resolved		awight	T140515 Donations message with empty body!
		Open		None	T140564 Prevent bad Cyrillic encoding

Event Timeline

awight created this task.Jul 15 2016, 8:38 PM

Restricted Application added subscribers: Zppix, Aklapper. · View Herald TranscriptJul 15 2016, 8:38 PM

We should test for this condition and throw a reasonable error.

Another one, this was not redelivered:

20:41:47      [headers] => Array
20:41:47          (
20:41:47              [gateway] => globalcollect
20:41:47              [source_enqueued_time] => 1468613976
20:41:47              [message-id] =>
20:41:47  ID:silicon-54591-1463622100074-2:1676475:-1:1:1
20:41:47              [source_type] => payments
20:41:47              [source_host] => payments1003
20:41:47              [destination] => /queue/donations
20:41:47              [timestamp] => 1468613976938
20:41:47              [expires] => 0
20:41:47              [persistent] => true
20:41:47              [source_run_id] => 26984
20:41:47              [priority] => 4
20:41:47              [php-message-class] =>
20:41:47  SmashPig\CrmLink\Messages\DonationInterfaceMessage
20:41:47              [source_name] => DonationInterface
20:41:47              [source_version] =>
20:41:47  8bf6e911eb43a2d369bf656f07d1b51be0a54f6c
20:41:47              [correlation-id] => globalcollect-8604182787
20:41:47          )

Change 299261 had a related patch set uploaded (by Awight):
Fail if the message body is missing

https://gerrit.wikimedia.org/r/299261

gerritbot added a project: Patch-For-Review.Jul 15 2016, 8:46 PM

awight triaged this task as Unbreak Now! priority.Jul 15 2016, 8:49 PM

Restricted Application added subscribers: Luke081515, TerraCodes. · View Herald TranscriptJul 15 2016, 8:49 PM

Change 299261 merged by jenkins-bot:
Fail if the message body is missing

https://gerrit.wikimedia.org/r/299261

The robust consumer patch is deployed now, so future empty bodies will cause a failmail with the message "Bad news, this message was lacking a body."

Ethereal!

Now we need to track down the source.

awight lowered the priority of this task from Unbreak Now! to High.Jul 15 2016, 9:19 PM

awight moved this task from Doing to Backlog on the Fundraising Sprint Muggle Baiting board.Jul 15 2016, 11:33 PM

I've been using this query to check Ingenico volume:

select count(*) from civicrm_contribution where receive_date > '2016-07-15 20:17:00' and trxn_id like 'GLOBALCOLLECT%';

also--the #signal-to-noise on our failmails is bad. In the future, we should be able to acknowledge the alarm and have a light quietly flash red...

Change 299280 had a related patch set uploaded (by Awight):
Log completed donations

https://gerrit.wikimedia.org/r/299280

This patch could use some review... I've included a test to make it more palatable... https://gerrit.wikimedia.org/r/299280

MBeat33 subscribed.Jul 16 2016, 5:29 PM

Change 299280 merged by jenkins-bot:
Log completed donations

https://gerrit.wikimedia.org/r/299280

Change 299290 had a related patch set uploaded (by Awight):
Log completed donations

https://gerrit.wikimedia.org/r/299290

Change 299290 merged by jenkins-bot:
Log completed donations

https://gerrit.wikimedia.org/r/299290

ReleaseTaggerBot added a project: MW-1.28-release (WMF-deploy-2016-07-19_(1.28.0-wmf.11)).Jul 16 2016, 7:00 PM

awight mentioned this in T140540: Non-1.28 branches should not be tagged 1.28.0-wmf.XX projects by ReleaseTaggerBot (namely Fundraising deployment branches).Jul 16 2016, 7:12 PM

Well. The debug logging just captured:

Jul 16 18:55:37 payments1002 globalcollect_gateway: 37122334:3271638092 Completed donation:

With no data encoded by json_encode( $this->getData_Unstaged_Escaped() ). Not what I was expecting...

I think that means json_encode is actually failing to encode, because any other type of empty data would result in a string like "false".

Change 299297 had a related patch set uploaded (by Awight):
[WIP] Log json_encode errors

https://gerrit.wikimedia.org/r/299297

@MBeat33
I'm realizing that we can detect this error while the donor is still logged in--
Maybe we should display a thank-you page that carefully alerts the donor that *you have paid $XX.00*, but we are experiencing a short delay in getting you a confirmation and welcome letter thanking you for the donation. Please save a copy of this page for your records...