This might be an unusual use case, but I found myself running test_model on a huge data set, and discovered that it's single-threaded. It would be nice to have a parallel job option.
I took a look at this to see if it'd be easy, and I found that test() is built into Model. Unfortunately, that means that it doesn't benefit from the worldly threading stuff in ScoreProcessor. Intriguingly, Model#test is never overridden by any subclasses. This makes me think that we can split out model testing into its own component.
On a less dramatic note, it's not clear how to figure out a good default number of CPU workers for model testing. If we're calling from cv_train, we want just 1 test thread per fold, but for a standalone test_model we would want to default to the machine's number of cores. Perhaps our scoring tools should share a worker pool, and we give all CPU workers equal priority. This would let us queue up runs en masse, and make efficient use of the hardware.
@awight: good first bug tasks are self-contained, non-controversial issues with a clear approach and should be well-described with pointers to help the new contributor. Given the current short task description and your own comment in T169843#3410396 I'm removing the good first bug tag. Please re-add the tag once the task description has been polished and provides sufficient information for a new contributor. Thanks!