Page MenuHomePhabricator

Use interruptable parallelism
Open, Needs TriagePublic

Description

We currently use ThreadPoolExecutor to clone repos in parallel, but when a failure is encountered we still need to wait for all active threads to terminate naturally. If we switch to ProcessPoolExecutor instead, we can send kill signals to all active threads and cause the job to immediately quit.

Let's also keep this distinction in mind when building any other types of parallelism into Quibble.

Event Timeline

awight created this task.Oct 1 2019, 9:32 AM
hashar added a subscriber: hashar.Oct 2 2019, 6:06 PM

Thank you for the task! I can't remember the details, but although you can cancel a thread task, it is not actually going to be terminated. Maybe that is a limitation of how threads work?

Also have to remember to track the failures and prevent any other tasks to be executed:

That is currently implemented with a threading.Event() to synchronize the threads, which is then injected as a parameter to the worker function clone_worker(). If set, the task DOES run, but would just return early.

A better design would be to have the ProcessPoolExecutor to no more schedule any new task as soon as one failed (worked around with a bad design described above) AND do what this task is: terminate the remaining tasks instead of just waiting for them to finish.

Ref 1ac9158c4091b2d4f0a012d510b9c6986cbf43d9 for T233143

awight moved this task from Backlog to In progress on the Quibble board.Apr 19 2020, 8:03 AM

This will be implemented in https://gerrit.wikimedia.org/r/#/c/integration/quibble/+/587885/ , so moving this task to in-progress.