Page MenuHomePhabricator

Document existing queue usage
Closed, ResolvedPublic

Description

We should have an overview of the different ways we use queues, which systems and machines are involved and what happens to the data. Cover the edge cases, failure, requeueing. Starting point: https://www.mediawiki.org/wiki/Fundraising_tech/Message_queues

Needs diagrams, and clarity about which operations are performed by each node.

We need this to have conversations about how to replace ActiveMQ.

https://www.mediawiki.org/wiki/Fundraising_tech/Message_queues

Event Timeline

awight created this task.Dec 17 2015, 8:21 PM
awight raised the priority of this task from to Needs Triage.
awight updated the task description. (Show Details)
awight added subscribers: Jgreen, atgo, awight, Aklapper.
Restricted Application added a subscriber: StudiesWorld. · View Herald TranscriptDec 17 2015, 8:21 PM
Jgreen added a parent task: Restricted Task.Mar 4 2016, 2:02 AM
awight updated the task description. (Show Details)Mar 23 2016, 9:10 PM
awight set Security to None.

Some glitches came to light while reading the code:

  • Payments and SmashPig are configured to look at two different limbo queues. This means that Ingencico, non-CC donations might be falling into a vortex.
  • Suspicious that we need to query Civi to see if a transaction exists. It would be much simpler to always write the data. The exception is when deciding whether to take further action to authorize, etc.
  • Stop referring to the "failed" queue in SmashPig, it's unused.

More things to note:

  • All pending and limbo mechanisms should be standardized.
  • Job runners should be rewritten as stream processors.
  • Delayed requeueing is an odd affair. Maybe we should formalize this with a "delay until" Z-set, which is popped by a periodic job, that then requeues into the original queue? That way, queue consumers don't need to duplicate delay logic.
  • DiskFileDataStore needs read support.
awight moved this task from Review to Done on the Fundraising Sprint Freshmaking board.
awight closed this task as Resolved.Mar 30 2016, 6:07 PM
awight claimed this task.
awight updated the task description. (Show Details)
mmodell removed a subscriber: awight.Jun 22 2017, 9:42 PM