The current [[ https://www.mediawiki.org/wiki/Help:Content_translation/Translating/Translation_quality | system of limits ]] in Content Translation includes relevant considerations to encourage users to edit the initial automatic translations, but it also has important limitations. For example, any translation with less than 50 paragraphs can be published even if it includes 95% of unmodified MT.
When proposing improvements to the system or alternative approaches for how modifications are calculated (T251893) or the limits applied (T245840) it is hard to anticipate how well will those work for most translations in different languages. This ticket proposes to create an automatic process for evaluating a limit system against a collection of translation samples.
The idea is to have two sets of translation samples: One with good translations that are expected to be allowed to publish, and another one with low-quality translations that are expected for the tool to prevent their publishing.
The process would apply a given algorithm of the limit system to all those samples and inform how many of the samples in each category were allowed, warned and prevented from publishing.
The translation samples can use the same format used for the parallel corpora to make it possible to extract samples from it (after human confirmation of them being hi or low quality), or create them as test cases (editing intensively or lightly the MT). Some of those samples could be obtained also as a byproduct of the upcoming research on translation quality (T288012).