We currently use ThreadPoolExecutor to clone repos in parallel, but when a failure is encountered we still need to wait for all active threads to terminate naturally. If we switch to ProcessPoolExecutor instead, we can send kill signals to all active threads and cause the job to immediately quit.
Let's also keep this distinction in mind when building any other types of parallelism into Quibble.
Thank you for the task! I can't remember the details, but although you can cancel a thread task, it is not actually going to be terminated. Maybe that is a limitation of how threads work?
Also have to remember to track the failures and prevent any other tasks to be executed:
That is currently implemented with a threading.Event() to synchronize the threads, which is then injected as a parameter to the worker function clone_worker(). If set, the task DOES run, but would just return early.
A better design would be to have the ProcessPoolExecutor to no more schedule any new task as soon as one failed (worked around with a bad design described above) AND do what this task is: terminate the remaining tasks instead of just waiting for them to finish.
This will be implemented in https://gerrit.wikimedia.org/r/#/c/integration/quibble/+/587885/ , so moving this task to in-progress.