To efficiently use our training resources we need to know how much parallelism to use for each portion of the training. There are a few levels of parallelism:
Primarily this ticket is concerned with the first type, although some information about how things scale when more models are trained in parallel could also be useful.
| Subject | Repo | Branch | Lines +/- | |
|---|---|---|---|---|
| Update resource usage docs | search/MjoLniR | master | +89 -60 |
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Invalid | None | T174064 [FY 2017-18 Objective] Implement advanced search methodologies | |||
| Resolved | EBernhardson | T161632 [Epic] Improve search by researching and deploying machine learning to re-rank search results | |||
| Resolved | EBernhardson | T170009 Evaluate training speed and accuracy for 1M and 30M sample training sets with different worker counts |
With a 9 features and a 1.1M train set, 10M test set:
| n_workers | took(s) | cpu took | holdout-test-ndcg@10 |
| 1 | 134.8s | 134.8s | 0.8274 |
| 2 | 92.5s | 185s | 0.8283 |
| 3 | 84.8s | 254.4s | 0.8275 |
| 4 | 79.3s | 317.2s | 0.8280 |
| 5 | 91.4s | 457.0s | 0.8277 |
At least for this sample it looks like either 1 or 2 workers is all we need. A single worker is the most computationally efficient, and 4 workers is the most real-time efficient. Using 4 workers vs 2 needs 2.3x more resources to do the same thing though. Using 2 workers vs 1 is 1.4x less efficient use of resources, but 1.5x more time efficient.
Running into Hadoop limitations for testing the 30M counts - moving to backlog for now.
I seem to have misplaced the raw data used for this ... but to summarize:
If the data all fits on a single executor, that is the most efficient use of cluster resources. This may not be the fastest way to train individual models, but if we are doing hyperparameter tuning we are generally training many models in parallel, and the lowest total cpu time used per model comes from using a single executor.
Training speed vs core count looks to stay relatively flat up to about 6 cores. Less parallelism is again more efficient in terms of total efficiency of the cluster, but up to 6 cores has a very minimal decrease. After 6 cores the efficiency loss starts to increase at a greater rate.
Overall suggestions:
Other:
We should document this info somewhere...but no more work needs to be done on this at this time.
Change 387442 had a related patch set uploaded (by EBernhardson; owner: EBernhardson):
[search/MjoLniR@master] Update resource usage docs