[17:29:13] <halfak> OK so I have an idea that I'm going to brain about real quick. 
[17:29:31] <halfak> We should split test_stats from thresholds in revscoring. 
[17:30:39] <halfak> thresholds are proving very useful to the ERI development team (e.g. RoanKattouw) as we had expected.  Threshold statistics are currently included with basic stats (like ROC-AUC) in test_stats.
[17:31:30] <halfak> But thresholds look different.  For example, we'll configure many different "filter_rate_at_recall" stats because we'd like to set different thresholds at different recall levels. 
[17:32:04] <halfak> So I think we should create a new thing called a "threshold". 
[17:32:19] <RoanKattouw> Don't mind me bloating your Makefile with all sorts of different recall_at_precision() stats... ;)
[17:32:47] <halfak> :D  that's a thing too.  If we want that many stats, we should have them.  However, they shouldn't be considered bloating. 
[17:33:12] <RoanKattouw> Are you suggesting that the APIs should be separate so I can get thresholds without ROC-AUC noise and vice versa?
[17:33:13] <halfak> I don't mind them in the makefile, but I think the JSON response in ORES is overwhelming to read through and I'd rather it wasn't. 
[17:33:20] <RoanKattouw> Right
[17:33:27] <halfak> Vice versa. 
[17:33:42] <halfak> I want ROC-AUC to be easy to get separate from the thresholds. 
[17:34:19] <halfak> The cool thing about "thresholds" is that it makes sense to set them manually, or tie them to a statistics.  Or ask them to optimize some statistics. 
[17:35:04] <halfak> OK, idea 2.  There's a thing call "thresholds" in the fitness metric methods called "thresholds" as well. 
[17:35:21] <halfak> This stores a set of thresholds at which the model will make predictions with a few nice guarantees. 
[17:35:54] <RoanKattouw> What kind of guarantees?
[17:36:01] <RoanKattouw> (BTW, https://github.com/wiki-ai/editquality/pull/63 )
[17:36:19] <halfak> Not too long of a vector, but adequately covers the real values a model produces. 
[17:36:41] <halfak> This is complex because a model's score is not guaranteed to be normally or uniformly distributed. 
[17:36:49] <RoanKattouw> Right
[17:37:17] <halfak> So, we store all the reasonable thresholds in some sort of array along with all of the 4 interesting test statistics. 
[17:38:07] <halfak> So a user could optimize after-the-fact.
[17:38:54] * AndyRussG is now known as AndyRussG|bassoo
[17:39:06] <halfak> So at every threshold, we store precision, recall, and filter_rate
[17:39:17] <halfak> You optimize how you like with the client. 
[17:39:21] <RoanKattouw> Ooh, that sounds great
[17:39:23] <halfak> :) 
[17:39:38] <halfak> I think it'll make it all easier. 
[17:39:45] <RoanKattouw> Are you saying we'd get that info for every threshold at an interval like 0.1 or 0.05, subject to excluding ranges that the model doesn't really ever reach?
[17:40:08] <halfak> Basically, yeah. 
[17:40:16] <RoanKattouw> Oh that would be excellent
[17:40:38] <halfak> So, in the short term, RoanKattouw, I suggest you continue to do what you are doing. 
[17:40:43] <RoanKattouw> I would basically not have to bother you ever again, and would have enough granularity to do all sorts of things
[17:41:36] <halfak> \o/ better for everyone.  Except it's fun to have you join us in this channel. :D
[17:42:01] <halfak> Anyway, I think you shoudl continue to propose a mess of test stats and in the meantime, I'm going to try to put this idea together in some tasks. 
[17:42:07] <RoanKattouw> Excellent
...
[18:31:02] <RoanKattouw> halfak: Shorter term, you said you could rebuild the models with my new stats tonight, when would they finish building?
[18:32:14] <halfak> Maybe tomorrow. :)  Assuming we didn't mess anything up when updating the file in the meantime. :) 
[18:32:45] <halfak> Oh man.  That's another benefit.  We won't need to rebuild models to incorporate new test threshold-level statistcis.