Adding to @Catrope, we could add a step to the Makefile after writing model stats which would look for obvious shortcomings in the models built and do some type of intervention. I'm not sure it should be a prompt however, because we run this as an ad-hoc batch process and want it to complete without blocking on any one model. For reference, I think we currently crash the entire build on errors anywhere in the make pipeline, so the bar is set pretty low.

Just taking a look at this again. I can explain where the problem may have come from with fiwiki (using flaggedrevs as observations), but the others have surprised me. It could be that by re-tuning and re-training we can get a more reasonable split. It's really in that case that I'm seeing a *serious* problem.

Generally, it seems likely that we'll continue to sometimes be able to satisfy strict statistics and struggle at other times. This is due to non-deterministic effects in model training. In reality, the model will be a bit better than the statistics suggest. Our statistics will get more and more exact as we add new observations to training and testing. This is a big reason why we want to get Jade out. It will be a huge source of data beyond the limited Wikilabels campaigns we run now.

That said, for some of these communities, we're still working with data from 2015/2016 so running a new labeling campaign to get more data wouldn't be out of the question.