Page MenuHomePhabricator

Threshold rounding considered harmful
Open, NormalPublic

Description

In revscoring, ScoringModels have a list of thresholds and corresponding fitness statistics. These are used for threshold optimizations. In order to reduce the memory footprint of these lists, we round thresholds by 3 decimal points by default so that we'll have at most 999 thresholds to store in memory.

In some cases where models attain very high fitness, this is problematic. As confidence approaches one or zero (0.999 or 0.001 respectively), there are useful thresholds that are hidden behind the rounding.

In order to fix this issue, we should implement a more intelligent strategy for choosing *useful* thresholds at which to generate fitness statistics. @Catrope suggested we look towards adaptive super-sampling for inspiration. Essentially, we would look for values between thresholds that offer interesting distinctions in fitness measures and then dig in there.

Event Timeline

Halfak created this task.Jul 19 2018, 12:56 PM
Restricted Application added a project: artificial-intelligence. · View Herald TranscriptJul 19 2018, 12:56 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
Harej triaged this task as Normal priority.Apr 9 2019, 9:24 PM