Page MenuHomePhabricator

Document existing queue usage
Closed, ResolvedPublic


We should have an overview of the different ways we use queues, which systems and machines are involved and what happens to the data. Cover the edge cases, failure, requeueing. Starting point:

Needs diagrams, and clarity about which operations are performed by each node.

We need this to have conversations about how to replace ActiveMQ.

Event Timeline

awight raised the priority of this task from to Needs Triage.
awight updated the task description. (Show Details)
awight added subscribers: Jgreen, atgo, awight, Aklapper.
Jgreen added a parent task: Restricted Task.Mar 4 2016, 2:02 AM
awight set Security to None.

Some glitches came to light while reading the code:

  • Payments and SmashPig are configured to look at two different limbo queues. This means that Ingencico, non-CC donations might be falling into a vortex.
  • Suspicious that we need to query Civi to see if a transaction exists. It would be much simpler to always write the data. The exception is when deciding whether to take further action to authorize, etc.
  • Stop referring to the "failed" queue in SmashPig, it's unused.

More things to note:

  • All pending and limbo mechanisms should be standardized.
  • Job runners should be rewritten as stream processors.
  • Delayed requeueing is an odd affair. Maybe we should formalize this with a "delay until" Z-set, which is popped by a periodic job, that then requeues into the original queue? That way, queue consumers don't need to duplicate delay logic.
  • DiskFileDataStore needs read support.
awight claimed this task.
awight updated the task description. (Show Details)