Change 503559 had a related patch set uploaded (by Petar.petkovic; owner: Petar.petkovic):
[mediawiki/extensions/ContentTranslation@master] Introduce less strict threshold for suppressed MT issues

https://gerrit.wikimedia.org/r/503559

Change 503559 merged by Petar.petkovic:
[mediawiki/extensions/ContentTranslation@master] Introduce less strict threshold for suppressed MT issues

https://gerrit.wikimedia.org/r/503559

@Pginer-WMF - please review the following testing results:

Two parts of this tasks were tested separately:

• Lower thresholds for unmodified content
• investigate false positives in articles are in Category:Pages with unreviewed translations (and similar categories in other wikis)

I did testing on the following:

Consider a less strict threshold for paragraphs that users marked as resolved. For paragraphs where the unmodified content warning was shown but the user marked it as resolved, we can apply a less strict threshold (95% of Machine translation or 75% of source content).

Testing cases (the behavior could be specific to cx2-testing) showed that
(1) text translated with the MT translation option, when marked as resolved, is allowed to be published with 99% of unmodified text
(2) text translated "Copy original content" option, when marked as resolved, is allowed to be published with 99% of unmodified text. There might be an addtional warning

(3) there is a discrepancy between counting overall amount of unmodified text (an error) and paragraph-related warning of unmodified text reported as T225088: Discrepancy in percentage count of unmodified content between error and warning reports

Looking in db for enwiki, the number of the pages in the category Pages with unreviewed translations is small comparing to the number of articles that have tags contenttranslation, contenttranslation-needcheck, contenttranslation-v2 (even if we filter out contenttranslation-needcheck and take into account that two tags - contenttranslation-needcheck, contenttranslation-v2 - are assigned together for the translations made with v2:

mysql:research@dbstore1003.eqiad.wmnet [enwiki]> select ct_tag_id, ctd_name, ctd_count  from change_tag join change_tag_def on ct_tag_id=ctd_id where ct_tag_id in (41,169, 522) group by ct_tag_id;
+-----------+------------------------------+-----------+
| ct_tag_id | ctd_name                     | ctd_count |
+-----------+------------------------------+-----------+
|        41 | contenttranslation           |     12378 |
|       169 | contenttranslation-needcheck |       573 |
|       522 | contenttranslation-v2        |      1628 |
+-----------+------------------------------+-----------+
3 rows in set (0.01 sec)

mysql:research@dbstore1003.eqiad.wmnet [enwiki]>  select count(*) from categorylinks where cl_to='Pages_with_unreviewed_translations';
+----------+
| count(*) |
+----------+
|      187 |
+----------+
1 row in set (0.01 sec)

Open questions

• It might be that Pages with unreviewed translations category (and similar categories in other wikis) has many false positives. It's possible to test that in cx2-testing which does have the page Category:Pages with unreviewed translations. Should we come up with some testing scenarios to check that? What logic is behind putting a translated article into Pages with unreviewed translations category?
• should we check the number of pages in Pages with unreviewed translations category in different wikis?
• Also, ORES scores can be checked for the articles in Pages with unreviewed translations category

In T217653#5239081, @Etonkovidova wrote:

mysql:research@dbstore1003.eqiad.wmnet [enwiki]>  select count(*) from categorylinks where cl_to='Pages_with_unreviewed_translations';
+----------+
| count(*) |
+----------+
|      187 |
+----------+
1 row in set (0.01 sec)

I'm curious about this query. For English it returns 187 results which match the total number of articles displayed in the category itself and is consistent with a query I made to check the distribution in months.
However, when checking it for spanish see only one result with the query but 2482 in the category. Do you know what exactly is listed in the categorylinks table and why this inconsistency?

Open questions

• It might be that Pages with unreviewed translations category (and similar categories in other wikis) has many false positives. It's possible to test that in cx2-testing which does have the page Category:Pages with unreviewed translations. Should we come up with some testing scenarios to check that?

I have the impression that there may be still a high number of high positives, but most of them are hard to test in an automatic way. There are the following kinds of problems:

• The user does not modify a paragraph because the content is not expected to be edited at all, but the tool does not know about it (e.g., math formulas: T225118).
• Machine translation worked so well for a paragraph that the user had to edit it much less than expected, and the tol thinks the content is not edited enough.
• The user makes enough modifications to the content but our code to identify those is not counting them.

From those, the last one is the one that seems easier to test automatically, but I don't expect to represent most of the problematic cases.

What logic is behind putting a translated article into Pages with unreviewed translations category?

A quick summary would be that translations are added when they are published but they still have 2 to 9 paragraphs with unmodified contents (each with +80% of the original MT, or +60% of the original text). These values change based on whether the user confirmed the warnings or whether the user had previously deleted translations. This is described in the documentation about the limits.

• should we check the number of pages in Pages with unreviewed translations category in different wikis?

Yes, especially given the inconsistencies I mentioned above. It would be useful to consider some of the top translated languages (French, Spanish, Catalan) as well as those not supporting MT (German and Italian).

• Also, ORES scores can be checked for the articles in Pages with unreviewed translations category

That can be interesting. I have not thought much yet on how the ORES predictions can be applied, but I'm open for suggestions.

In T217653#5240711, @Pginer-WMF wrote:

In T217653#5239081, @Etonkovidova wrote:

mysql:research@dbstore1003.eqiad.wmnet [enwiki]>  select count(*) from categorylinks where cl_to='Pages_with_unreviewed_translations';
+----------+
| count(*) |
+----------+
|      187 |
+----------+
1 row in set (0.01 sec)

I'm curious about this query. For English it returns 187 results which match the total number of articles displayed in the category itself and is consistent with a query I made to check the distribution in months.
However, when checking it for spanish see only one result with the query but 2482 in the category. Do you know what exactly is listed in the categorylinks table and why this inconsistency?

It turned out that Páginas_con_traducciones_sin_revisar category in eswiki is, in fact, Wikipedia:Páginas_con_traducciones_sin_revisar. When the category name is corrected, the query returns the number of articles that is close to the number lshown on Páginas con traducciones sin revisar page.

mysql:research@dbstore1003.eqiad.wmnet [eswiki]> select year(cl_timestamp) as Year,MONTHNAME(cl_timestamp) as Month, count(*) as Unreviewed 
    -> from categorylinks 
    -> where cl_to='Wikipedia:Páginas_con_traducciones_sin_revisar'
    -> group by year(cl_timestamp),month(cl_timestamp)
    -> order by cl_timestamp desc;
+------+----------+------------+
| Year | Month    | Unreviewed |
+------+----------+------------+
| 2019 | June     |         65 |
| 2019 | May      |        303 |
| 2019 | April    |        515 |
| 2019 | March    |        757 |
| 2019 | February |        495 |
| 2019 | January  |        360 |
+------+----------+------------+
6 rows in set (0.01 sec)

Thanks, @Pginer-WMF for your comments. It seems that the mechanism of putting a translated article into "Pages with unreviewed translations" category (and equivalent categories in other wikis) is well defined and documented. If a clear logic for assigning translated articles is in place, then I don't have any questions regarding actual assignment to the category. Unless there are some doubts about whether that logic is correctly applied (that could be tested).

To test the validity of the logic (i.e. how the quality of article suffers with actual tresholds for unmodified content) is, as you mentioned it, not quite possible. It might be interesting to get stats on how ORES scores articles that have contenttranslation tags.

Another thought on false positives in "Pages with unreviewed translations" category - what do we know about how editors remove such category? It just might be that the category simply was not been removed after an article had been improved.

Thanks @Etonkovidova for sharing your thoughts and helping with the query.
Regarding ORES, I made a quick experiment with Recent Changes and there is not much diversity in scores, all published translations are classified as good and made in good faith. (cc @Halfak )

Regarding categories not being removed after the translation is review is totally possible. For that, I guess that communication to get editors aware of the category can help.

@Etonkovidova, do you need any additional info to continue QA on this ticket?

The criteria for preventing user from publishing or adding to tracking category is documented in this mediawiki page. I suggest you cover all the cases laid out in T221359, while paying attention that MT limits vary if user marks the MT abuse warning as resolved, which is the topic of this ticket.

Also, when checking MT percentage on paragraph level, keep in mind the numbers are not updated as you type, but after some delay, which is covered in T200683.

@Etonkovidova I instructed you to cover cases laid out in T221359 and try having some sections with high MT warnings marked as resolved, because that is when MT limits change. In the end, that is what this ticket was about.

As for limits, T221359#5258405 changed the spec.

In T217653#5258417, @Etonkovidova wrote:

To check the tests for a user with previous deleted translations, I'll need user rights (in cx2-testing) to be able to delete articles.

@KartikMistry, please add necessary rights to @Etonkovidova's test accounts on cx2-testing for this.

In T217653#5258467, @Petar.petkovic wrote:

@Etonkovidova I instructed you to cover cases laid out in T221359 and try having some sections with high MT warnings marked as resolved, because that is when MT limits change. In the end, that is what this ticket was about.

It's been part of the testing

• to see what effect checking 'Resolve' or 'Publish anyway' (to publish with existing issues) has on actual publishing.
• I varied the amount of MT (and unmodified content) - from 70% to 98% when I was checking the above.
• checked the calculation of MT content percentage (counting chars and substitute specific amount with modified content) - it was done previously
• compared the paragraph to paragraph MT calculation to the progress bar calculation (also done in other ticket).

Notes for testing:

• two users: Administrator and extendedconfirmed
• extendedconfirmed user publishes couple of translations which will be deleted by an Administrator
• check whether extendedconfirmed user would be prevented to publish according to the limitations in T221359.