Page MenuHomePhabricator

Quit looking in pending queue for completion message information
Closed, ResolvedPublic2 Story Points

Description

We're processing Amazon messages in a fragile way. We send a message from DonationInterface to the pending queue with donor information and utm data. Then when the IPN message comes in, we send a sparse message to the donations queue with the amount, time, and ID of the message we assume is in the pending queue.

When the donations queue consumer sees a message with a completion_message_id, it tries to find that message in the pending queue. If it doesn't exist, it takes 5 seconds to time out, which can back the queue up like crazy if there are a lot of missing messages in the pending queue.

We should store the pending info someplace where it's a lot easier to look things up by ID and not as expensive when it fails.

Details

Related Gerrit Patches:
wikimedia/fundraising/SmashPig : masterStop using pending queue in RecordCaptureJob
wikimedia/fundraising/SmashPig : masterStop using pending queue in ProcessCaptureRequestJob
wikimedia/fundraising/crm : masterDon't pollute $msg with pending_id
wikimedia/fundraising/crm : masterCheck for duplicates before re-queueing MISSING_PREDECESSOR
wikimedia/fundraising/crm : masterImport info from pending DB, not AMQ
wikimedia/fundraising/crm : masterWIP tests for combining pending DB info

Event Timeline

Ejegg created this task.Dec 30 2015, 4:35 AM
Ejegg raised the priority of this task from to High.
Ejegg updated the task description. (Show Details)
Ejegg added a subscriber: Ejegg.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptDec 30 2015, 4:35 AM
awight added a subscriber: awight.
DStrine set Security to None.Jan 19 2016, 8:36 PM
DStrine edited a custom field.

Change 299938 had a related patch set uploaded (by Ejegg):
DO NOT MERGE: import info from pending DB, not AMQ

https://gerrit.wikimedia.org/r/299938

Ejegg moved this task from Backlog to Doing on the Fundraising Sprint Nitpicking board.

Change 300584 had a related patch set uploaded (by Ejegg):
WIP tests for combining pending DB info

https://gerrit.wikimedia.org/r/300584

Change 301313 had a related patch set uploaded (by Ejegg):
DO NOT MERGE: kill SmashPig pending queue code

https://gerrit.wikimedia.org/r/301313

Change 305577 had a related patch set uploaded (by Ejegg):
Check for duplicates before re-queueing MISSING_PREDECESSOR

https://gerrit.wikimedia.org/r/305577

Change 300584 abandoned by Ejegg:
WIP tests for combining pending DB info

Reason:
Folded in with I29477a5319180d54

https://gerrit.wikimedia.org/r/300584

Change 307359 had a related patch set uploaded (by Ejegg):
Stop using pending queue in ProcessCaptureRequestJob

https://gerrit.wikimedia.org/r/307359

Change 299938 merged by jenkins-bot:
Import info from pending DB, not AMQ

https://gerrit.wikimedia.org/r/299938

Change 305577 merged by Ejegg:
Check for duplicates before re-queueing MISSING_PREDECESSOR

https://gerrit.wikimedia.org/r/305577

Change 307445 had a related patch set uploaded (by Ejegg):
Don't pollute $msg with pending_id

https://gerrit.wikimedia.org/r/307445

Change 307445 merged by jenkins-bot:
Don't pollute $msg with pending_id

https://gerrit.wikimedia.org/r/307445

Change 307359 merged by jenkins-bot:
Stop using pending queue in ProcessCaptureRequestJob

https://gerrit.wikimedia.org/r/307359

Change 301313 merged by jenkins-bot:
Stop using pending queue in RecordCaptureJob

https://gerrit.wikimedia.org/r/301313

Ejegg closed this task as Resolved.Sep 13 2016, 10:14 PM
mmodell removed a subscriber: awight.Jun 22 2017, 9:44 PM