Page MenuHomePhabricator

Re-run the MT service usage report after MinT is made available to a broad set of languages
Closed, ResolvedPublic

Description

After the two previous reports about machine translation usage (T303812 , T310773), there has been significant change in machine translation support. The introduction of MinT exposed several translation models such as NLLB-200 and Opus to many languages: several languages getting machine translation support for the first time, others getting an alternative that may or may not be preferred to the existing alternatives.

Once the process of enabling languages has stabilized, we may want to re-run the report. In this way we can understand which services are used (consider adjust default options) and have a sense of the impact in terms of edits to the initial translation and deletion rates.

Codes for the supported languages with MinT are available in this configuration file.

Result


The results of this ticket will inform the change of defaults: T341458: Set MinT as the default for languages where it is optional but frequently used

Event Timeline

Once the enablements from T340953 and T333969 are completed, we can wait two months and re-run the report.

Pginer-WMF raised the priority of this task from Medium to High.Oct 2 2023, 9:31 AM

@Pginer-WMF We spoke to analysing the data from the last two months, which is fine for most of the report. But the previous report has section where the usage stats were compared pre/post some changes (new language additions and changes to default service). I am wondering if you would be interested in something similar for this report as well, especially as the focus is on MinT. If yes, is there a date that can be used as reference point to compare pre/post?

@Pginer-WMF We spoke to analysing the data from the last two months, which is fine for most of the report. But the previous report has section where the usage stats were compared pre/post some changes (new language additions and changes to default service). I am wondering if you would be interested in something similar for this report as well, especially as the focus is on MinT. If yes, is there a date that can be used as reference point to compare pre/post?

The previous report has a "Changes in machine translation usage since the first report" section. A similar comparison can be useful to have a sense of how the usage of the different options evolve as new services and more languages are supported. For example, as MinT supports more languages the expectation is for the percentage of translations started from scratch to be reduced, and it would be interesting to check how much.

The recent expansion period where MinT was enabled for hundreds of languages was between May and July 2023. So I think it makes sense to compare how the usage was distributed before and after the period. For the "after" period I think it makes sense to use the range we are usign for the overall report (August - September 2023). For the "before" period I can think of two options:

  • A period of similar length immediately before the recent enblements started (March - April 2023)
  • Have a Year-ove-Year (YoY) comparison (August - September 2022) since that is a period that takes pace after the enablements of the 2nd report and before the recent set of enablements.

I'd be inclined to the YoY comparison since it has the benefit of accounting for seasonality, but happy to go with a more recent period (or something different) if you think it is better.

@Pginer-WMF the report is complete, this is the new link.

One consideration for the "scratch" concept, is that we may want to distinguish two scenarios:

  • Starting from scratch when there are other MT service. this represents a signal of the MT services being of low quality and a preference for the user.
  • Starting from scratch when there is no other option. This just reflects the lack of availability of MT options. It can be still useful to identify which languages can benefit the most from getting MT, but it does not capture a user preference since there were no options. For example, when translating to English, German and Japanese, machine translation is not enabled in content translation.

Having a break down for the current "scratch" numbers could be a useful consideration for the next re-run

Another consideration for the next re-run based on the feedback we received: It may be relevant to incorporate the perspective of user expertise. Are newcomers/experts using one service more, are their translations deleted more often?
For example, on wikis where MinT is provided as optional, maybe experienced editors are the ones more prone to find (and willing to try) the option.

I'm not sure which is the best way to capture this to have useful information while not fragmenting the data representations too much. We can talk more on this for the next iteration.