Change 476480 had a related patch set uploaded (by Santhosh; owner: Santhosh):
[mediawiki/extensions/ContentTranslation@master] Add a counter for published translations with unreviewd MT

https://gerrit.wikimedia.org/r/476480

Change 476480 merged by jenkins-bot:
[mediawiki/extensions/ContentTranslation@master] Add a counter for published translations with unreviewed MT

https://gerrit.wikimedia.org/r/476480

Petar.petkovic moved this task from In Review to QA on the Language-Team (Language-2018-October-December) board.

@Petar.petkovic, I don't see the new line in the graph. Is this waiting for deployment or is further development needed?

I checked today (the task is on my list of tasks to check after wmf.8 deployment) - the link https://language-reportcard.wmflabs.org/cx2/scripts.js?v=abce880:46 displays the error in the Console:

Wed, 12 Dec 2018 23:23:12 GMT https://language-reportcard.wmflabs.org/cx2/scripts.js?v=abce880:46

TypeError: e is undefined scripts.js:36:7482

The link was working last week.

This is what I think happened: @Amire80 updated the tab names, so that the new link is https://language-reportcard.wmflabs.org/cx2/#cx-2-translations (cf. https://language-reportcard.wmflabs.org/cx2/#translations) but the link in the task description was not updated.

Thx, @Nikerabbit - https://language-reportcard.wmflabs.org/cx2/#cx-2-translations is working, but the new graph "Published needing review" is still not present.

Given that this may take some more time due to technical complexities, I created a ticket for an initial report that will provide some initial insights and useful data for upcoming communications: T218020: Measure percentage of translations published with and without the expected level of modified content

Change 502312 had a related patch set uploaded (by Amire80; owner: Amire80):
[mediawiki/extensions/ContentTranslation@master] Log need-review events

https://gerrit.wikimedia.org/r/502312

Change 502312 merged by jenkins-bot:
[mediawiki/extensions/ContentTranslation@master] Log need-review events

https://gerrit.wikimedia.org/r/502312

So I have the events being logged, and to show them on the chart, I'll have to do something like this regularly:

select
  count(distinct(concat_ws(',', event.sourceTitle, event.sourceLanguage, event.targetTitle, event.targetLanguage, event.token))) as count
from
  event.contenttranslation
where
  event.action = 'need-review' and
  year = 2019 and
  month = 5 and
  day = 3;

This will have to be in hive. As a first step, I'll probably make a new dashboard for this, and once it works, I'll merge the existing dashboard into it.

A question for @mforns: The query above selects just one number, per day. If I understand correctly, the output is supposed to include two columns: a date and a number. If I'm going to use RU, what's the right way to add the number? And what's the most robust way to pass this date to the query in a way that will show it as a column, and will break it correctly for the year/month/day conditions?

@Amire80
When we use RU for Hive, we have to use a script instead of the query.
That is so, because RU doesn't have yet a Hive client. So we use a bash script that calls hive -e "<query>".
The way RU passes dates (and other params) to the script is different from the way it passes dates to sql files.
In a nutshell, to add a date column in a Hive query (bash script) use:

SELECT
    ...
    '$1' AS date,
    ...

$1 is the first parameter that RU passes to the script, which is the date in question.
You can find this and other infos in the RU documentation:
https://wikitech.wikimedia.org/wiki/Analytics/Systems/Reportupdater
Also, take a look at this example of another Hive-based RU report:
https://github.com/wikimedia/analytics-limn-language-data/blob/master/interlanguage/percent_interlanguage_navigation_curr
You can basically copy the way hive is called (hive -e "..." 2> /dev/null | grep -v parquet.hadoop).
And also, copy the way $1 is used.

I eventually decided to do this in mysql, and perhaps later move everything to hive if it's desirable. This is supposed to make the initial deployment easier.

Change 509007 had a related patch set uploaded (by Amire80; owner: Amire80):
[analytics/limn-language-data@master] Add need-review chart to published CX2 translation

https://gerrit.wikimedia.org/r/509007

Change 509007 merged by jenkins-bot:
[analytics/limn-language-data@master] Add need-review chart to published CX2 translation

https://gerrit.wikimedia.org/r/509007

OK, so now thanks to @mforns the chart works at https://language-reportcard.wmflabs.org/cx2/#cx-2-translations-that-need-review . However, this task ask to put the chart at the "CX2 translations that need review" tab together with the "CX2 translations" tab, as just one tab with two lines. I guess that this may be possible by editing the JSON configuration at https://meta.wikimedia.org/wiki/Config:Dashiki:CX2Translations , but I'm not sure how exactly.

Relevant update at T210138#5256469.

We may want to review the task request, probably incorporating the new data in the new Superset dashboard.
We may consider deprecating the specific CX2 tags since that is now the only version (no longer coexisting with CX1).

@Pginer-WMF After exploring the sources available, here are my initial findings:

• There seems to be no direct way to count translations marked for review, yet.
• The EventLogging schema for ContentTranslation has a need-review value for the action field, however, this schema is no longer in use. The new schema hasn't been fully instrumented (note, there is no tag for events that need review in the new schema).
  • With the new schema, if human_modification_rate is less than human_modification_threshold and the translation has been published, we can potentially consider it as a translation that needed review. We will still be missing historical data.
• There is a maintenance category being added to all translations that need review (T190798). Using the categorylinks table we can use this to get a live count of translations that are yet to be reviewed, and cl_timestamp can give the information about when the articles were added to the category. However, this may not be very accurate to track historically because if an article has been reviewed and the category was removed, it will be missed in the historical counts.
• We can potentially use the translation_progress field in the cx_translations table as it captures the data for unmodified machine translation content.
  • However, I am not very sure it this is accurate enough. While exploring the data, I found some exceptions:
    • For example, Mercedes Baptista article on enwiki was marked for review, even though unmodified mt content is only 4%
  • I have explored the data for five wikis (en, fr, hi, he, it), and the average percentage of mt content for pages in the need to be reviewed category is ~40% (but also high std deviation ~30%)
  • If we have thresholds defined by wiki, we can use the data from the cx_translations table to identify translations marked for review.

But in the long term, I think having a tag either in the schema or as an edit tag will be better.

Thanks for the analysis @KCVelaga_WMF

We have plans to implement the instrumentation for the publishing step with the new schema (T318906). If there are particular requirements that are worth considering we can incorporate them in the ticket to make sure those are supported.

Similarly we can consider adjustments on the tags used when articles are published.

In T209868#8984707, @KCVelaga_WMF wrote:

• There is a maintenance category being added to all translations that need review (T190798). Using the categorylinks table we can use this to get a live count of translations that are yet to be reviewed, and cl_timestamp can give the information about when the articles were added to the category. However, this may not be very accurate to track historically because if an article has been reviewed and the category was removed, it will be missed in the historical counts.

An important consideration is whether we aim for measuring (a) how many articles had a high percentage of MT at the time of publishing or (b) how many articles require review (those still in the category, but not those no longer there).

Based on your comments, the current category could support (b) and an edit tag would be needed for (a).

Ideally it would be great to know how many translations were published with too much MT, and which percentage of those went out of the category because they were good or improved to be good and which were deleted.

Removing unrelated account @KCVelaga

@Pginer-WMF

Sure, @MNeisler mentioned plans to add more granular tags with the schema possibly. We can discuss further on that.

Yes, from the current category available, we would know how many articles need review, at the present.

Having an edit tag will help us understand both

• trends of articles published with high MT content (and tagged for review on the respective wikis)
• combine that with the category counts to see how many of them were improved and delete

An edit tag will be helpful for the scope of this task. Having an edit tag will make it easy to integrate these metrics into the current Superset dashboard, as we will be able to query the data from edit_hourly, like the rest of the charts.

I see that this approach was considered earlier: T190798, but dropped in favour of the maintenance category. I will go ahead and reopen the task.

@Pginer-WMF

As revision tag ( T160037) is now enabled, I have added a chart to the dashboard to show the translations tagged with high unmodified machine translated text, as percentage of published translations.

Please review and let me know what you think.