First up: Train the same data with the same hyperparameter space multiple (3) times to get an idea of expected variance

5 would have been nice, but even 3 took a significant amount of resources (which unfortunately collided with the end of month jobs).

min diff: 0.0000
max diff: 0.0016
mean diff: 0.0005

This isn't too far off from a list of wikis ordered by number of available observations. It seems like on the largest wikis improvements of 0.0005 or more might be something, where on smaller wikis we might want to see 0.001 or 0.0015. These are actually a good bit smaller than I was expecting

Next up: Experiment with variations in number of trees for small wikis, does current setting of 500 help?

Ran 400 iterations of hyperopt against kowiki (~562k observations, on the small side) with number of leafs from 8000 to 64000 along with the normal hyperparameters, and setting number of iterations based on leafs >> max_depth. Best cv-test-ndcg@10 of 0.8497. Previous best without tuning leaf counts was 0.8493. For the real kicker, this was achieved with number of leafs of 8014 and max_depth of 4, which works out to 500 trees and exactly how many was arbitrarily chosen before this test. That bumps right up into the lower limit so it might be worth searching a little more, but i expect improvement would still be minimal.

This does lead me to a question though, should we add ndcg@3 and ndcg@1 metrics to standard evaluations? It's possible there is improvement that is limited to only the top results?

Overall this doesn't seem to need further investigation. It may be a minor optimization but probably not worth spending much effort or sacrificing training speed for.

Next up: Train with more hyperopt iterations (300? 500?), to see if continued search is beneficial.

Didn't directly test this, but dug through previous records of training runs. We typically do 150 iterations of hyperopt and I'm seeing on average 15 iterations that report cv-test-ndcg@10 values within 0.0010 of the best value. This suggests to me that the cv-test-ndcg@10 found is bumping into the limits of the current training data and we could even consider dropping the iteration count to 80 or 100. Additionally i ran the above test (tuning by leafs) for 400 iterations and while the value it found was slightly higher, it wasn't high enough to be a significant result.

Adding more features to the feature set may change this, and we should revisit once we add new features and have a few runs worth of training hitsory to evaluate.

My suggestion here is that perhaps we *lower* the number of iterations, as we seem to be doing more iterations than necessary. Perhaps useful would be to add a utilities script to mjolnir to walk through a directory containing training result sub directories to report on stats across runs.