Page MenuHomePhabricator

Process Adyen IPN messages in parallel (improve TY delivery times)
Closed, ResolvedPublic


We currently put all Adyen IPN messages onto the same queue as they come in, then process them one at a time. This process takes a few seconds for each one since we make a couple of API calls. Unfortunately, that's not fast enough to keep up with our generous donors when we set Adyen up to receive half of the traffic from US banners. This morning's test saw a constantly-growing Adyen queue, which only started dropping when we turned banners off. Donor TY mails were up to 1 hr late after three hours of the test.

Event Timeline

DStrine renamed this task from Process Adyen IPN messages in parallel to Process Adyen IPN messages in parallel (improve TY delivery times).May 6 2020, 3:54 PM
DStrine moved this task from Q3 2021-2022 to Triage on the Fundraising-Backlog board.

It would be nice to have a generalized way to multiplex any of the queues defined in our config yaml files.

I'm concerned that with scaling up transaction volume (if we go w/Adyen for EN6C cards for example) we might see an increase in donors asking if we received their successful donations. If we see that in Q2 I might ask to prioritize this task higher.

DStrine raised the priority of this task from High to Unbreak Now!.Dec 4 2020, 3:51 PM

Change 645397 had a related patch set uploaded (by Ejegg; owner: Ejegg):
[wikimedia/fundraising/SmashPig@master] WIP Adyen job queue fan-out

Change 645397 merged by jenkins-bot:
[wikimedia/fundraising/SmashPig@master] Adyen capture job queue multiplexing

Code is deployed, but we need to deploy some config changes

  • Add definitions for jobs-adyen-1, jobs-adyen-2, and jobs-adyen-3 queues to smashpig/main.yaml (in puppet)
  • Create process-control jobs to consume from the new queues
  • Update SmashPig/local-config/adyen/main.yaml to set capture-job-queue-count: 3