Page MenuHomePhabricator

Document existing queue usage
Closed, ResolvedPublic


We should have an overview of the different ways we use queues, which systems and machines are involved and what happens to the data. Cover the edge cases, failure, requeueing. Starting point:

Needs diagrams, and clarity about which operations are performed by each node.

We need this to have conversations about how to replace ActiveMQ.

Event Timeline

awight created this task.Dec 17 2015, 8:21 PM
awight raised the priority of this task from to Needs Triage.
awight updated the task description. (Show Details)
awight added subscribers: Jgreen, atgo, awight, Aklapper.
Restricted Application added a subscriber: StudiesWorld. · View Herald TranscriptDec 17 2015, 8:21 PM
Jgreen added a parent task: Restricted Task.Mar 4 2016, 2:02 AM
awight updated the task description. (Show Details)Mar 23 2016, 9:10 PM
awight set Security to None.

Some glitches came to light while reading the code:

  • Payments and SmashPig are configured to look at two different limbo queues. This means that Ingencico, non-CC donations might be falling into a vortex.
  • Suspicious that we need to query Civi to see if a transaction exists. It would be much simpler to always write the data. The exception is when deciding whether to take further action to authorize, etc.
  • Stop referring to the "failed" queue in SmashPig, it's unused.

More things to note:

  • All pending and limbo mechanisms should be standardized.
  • Job runners should be rewritten as stream processors.
  • Delayed requeueing is an odd affair. Maybe we should formalize this with a "delay until" Z-set, which is popped by a periodic job, that then requeues into the original queue? That way, queue consumers don't need to duplicate delay logic.
  • DiskFileDataStore needs read support.
awight moved this task from Review to Done on the Fundraising Sprint Freshmaking board.
awight closed this task as Resolved.Mar 30 2016, 6:07 PM
awight claimed this task.
awight updated the task description. (Show Details)
mmodell removed a subscriber: awight.Jun 22 2017, 9:42 PM
Jgreen moved this task from Triage to Done on the fundraising-tech-ops board.Feb 13 2020, 7:42 PM