Full details on MediaWiki.

Summary:

• A per-wiki unknown n-gram penalty makes a small difference in overall F0.5 score for most corpora.
• The unknown n-gram penalty can't be optimized across all the corpora; they each want their own value.
  • The "proportional penalty" isn't terribly useful, either.
• It's not worth pursuing, but it's worth keeping in our bag of tricks should it be useful in the future.
• It's only a few lines of code, so I'm not going to clutter up TextCat with more stuff we aren't going to use.

@TJones, thanks for he explanation you gave me yesterday.
I consider this task done, a new one should probably be created to actually implement this idea in TextCat?

In T151230#2971808, @dcausse wrote:

@TJones, thanks for he explanation you gave me yesterday.

Very welcome! Thanks very much for looking over it.

I consider this task done, a new one should probably be created to actually implement this idea in TextCat?

Because it didn't have much useful effect—so we aren't going to use it—and the code it fairly trivial, I decided not to implement it. If anyone thinks it might be nice to have anyway, I can implement it anyway. I could also test it against the old production baseline just to see how useful it is without all the other improvements, which have pushed accuracy up so high that it's hard to make much more progress in many cases.

If I do implement it, I can do it on this task, as I've done with the others. (This is the first one that didn't show enough improvement to implement and use.)