Page MenuHomePhabricator

process-control will need to run some jobs sequentially
Closed, ResolvedPublic2 Estimated Story Points

Description

For example, an audit fetch is immediately followed by audit parse, as long as the fetch was successful.

Let's consider doing this at the application level rather than in process-control?

Chained jobs

Amazon nightly audit download -> Amazon nightly audit parser
AstroPay nightly audit download -> AstroPay nightly audit parser
Fundraiser Public Data Export -> Samarium Data Mover
GlobalCollect audit file download -> Globalcollect WR1 Log Audit
PayPal nightly audit download -> PayPal nightly audit parser
Donations queue consume -> Thank you mail send (shouldn't do this)
Silverpop emails - Build export files -> Silverpop emails - Upload files

Event Timeline

ggellerman set the point value for this task to 2.

Change 345429 had a related patch set uploaded (by Awight):
[wikimedia/fundraising/process-control@master] Run commands in sequence

https://gerrit.wikimedia.org/r/345429

Change 345429 merged by jenkins-bot:
[wikimedia/fundraising/process-control@master] Run commands in sequence

https://gerrit.wikimedia.org/r/345429

awight removed a project: Patch-For-Review.
awight lowered the priority of this task from High to Medium.

I realize now that my solution was too naive. Specifically, staggering the queue consumer and thank-you jobs using the cheesy consume_and_thank parent job paradigm prevents reentry of the entire chain, when it should only lock sub-jobs. In other words, we prevent the next queue consumer from running if the previous thank-you job is still going, when we should let them overlap instead.

One simple fix might be to introduce a new property "reentrant: true" on the parent job, but continue to lock the sub-jobs to prevent two consumers or two thank-you mailers.

With consume/thank you, I think it's actually good to have each block the other, since they compete for the same tables. We just need the thank you job to have a time limit (T163412) so the total time never exceeds the duty cycle of 2 minutes.

Ejegg claimed this task.