Note that this will require a new event_source value in the content_translation_event schema (T287403).

I am generally not against the idea, but I have a bunch of comments about the proposal.

First, it should be treated as an experiment at first, and not immediately as a feature. If the experiment is successful, it can become a full-fledged feature. The things to test in the experiment are:

1. Do all the people really understand that this is an automatic suggestion and not a part of the article that was written by human Wikipedians? This must be tested while keeping in mind earlier research that showed that different people perceive unedited machine translation differently: some think that it's problematic and unusable without editing, and others think that it's mostly useful. However, our design solution must work for everyone: we must make it clear that it was not written by humans even to those who think that machine translation is OK.
2. Is it effective in actually converting people to translators? Doesn't have to be 100%, of course, but under a certain threshold it's probably not worth deploying. What's the threshold? I'm not sure; I could say "10%", but it's just a random proposal.
3. Are the translations created as a result of this actually good? It's difficult to measure with full automation, but we at least need to measure how much do people actually edit the machine translations. If they edit them very little or not at all, it's probably a failure.

The experiment must succeed on all three points. #1 may appear to be unimportant if #2 and #3 are successful, but actually, it is important, too: the readers, who are the mostly-silent majority, must not get machine-translated content and think it was written by humans. Also, keep in mind that this will disproportionately affect the smaller languages: readers in the largest languages, like English, French, and Russian will almost never see it, while readers in Bengali and other less-represented languages will see it very often.

The precise metric goals for #2 and #3 can be figured out later, but they must be tested.

I am concerned that some people may simply think that this is the content of the article. The proposal says "Make it clear that the translation is automatic and not part of the community-created content", but I strongly suspect that the current design doesn't make it clear enough.

The screenshots here do have some good parts:

1. The text of "Review and fix" puts the emphasis on reviewing and fixing.
2. It all comes after the end of the article (or, at least, what experienced Wikipedians like me experience as "the end of the article"...)

But they are also misleading in several ways:

1. The user interface is only partly translated. In the Bengali Wikipedia, the user interface parts must be translated to Bengali.
2. The "Review and fix" and the "Automatic translations are not always accurate" parts are small, gray, and too easy to miss.

The working language in the design and development team is English, so it will be good to turn the prototypes around and use English as the target language. This will be the best thing for understanding the target user experience: the user interface will be readable to the whole team, and the imperfections of the machine translation will be constantly seen by everyone, too.

In T270714#7334353, @Amire80 wrote:

I am generally not against the idea, but I have a bunch of comments about the proposal.

Thanks for the feedback, Amir. Some specific comments below.

First, it should be treated as an experiment at first, and not immediately as a feature. If the experiment is successful, it can become a full-fledged feature. The things to test in the experiment are:

1. Do all the people really understand that this is an automatic suggestion and not a part of the article that was written by human Wikipedians? This must be tested while keeping in mind earlier research that showed that different people perceive unedited machine translation differently: some think that it's problematic and unusable without editing, and others think that it's mostly useful. However, our design solution must work for everyone: we must make it clear that it was not written by humans even to those who think that machine translation is OK.

Agree. This is a key aspect we have iterated and I'm open to continue to do so. The ticket proposes a limited scope so that we can learn and adjust these aspects based on how they work with real content but limiting the exposure.

1. Is it effective in actually converting people to translators? Doesn't have to be 100%, of course, but under a certain threshold it's probably not worth deploying. What's the threshold? I'm not sure; I could say "10%", but it's just a random proposal.

I think that the percentage of conversion can be a bit misleading. I think we need to consider how many translations this entry point brings, how good these are, and verify that there is no negative impact on those not interested in translating (or even if there is a positive impact for them). But I think it is possible to succeed at all that even if the percentage of translators were small. Some example scenarios:

• This entry point brings a million users to translate successfully , more than any other entry point for it. Readers not interested in translating are not impacted negatively. However because the high number of visitors (from the 500 million users visiting Wikipedia each month) the actual conversion is a small percentage (0.2%). Why would be the small percentage a problem in this case?
• This entry point brings a small number of users to translate, but monolingual readers find it very convenient to learn more about topics and have a clear understanding that this is the same MT they could get through external tools (e.g., Google Translate website) but provided in a more convenient way (saving clicks, not requiring technical knowledge, without privacy concerns) that makes them learn more frequently from it. Assuming it is not creating confusion with community-created content, why should we ignore this kind of impact when evaluating the entry point?

1. Are the translations created as a result of this actually good? It's difficult to measure with full automation, but we at least need to measure how much do people actually edit the machine translations. If they edit them very little or not at all, it's probably a failure.

Makes sense. I think it would be useful to compare success across the translations made by the different entry points, based on edits and deletions, as well as human feedback.

I am concerned that some people may simply think that this is the content of the article. The proposal says "Make it clear that the translation is automatic and not part of the community-created content", but I strongly suspect that the current design doesn't make it clear enough.

The screenshots here do have some good parts:

1. The text of "Review and fix" puts the emphasis on reviewing and fixing.
2. It all comes after the end of the article (or, at least, what experienced Wikipedians like me experience as "the end of the article"...)

But they are also misleading in several ways:

1. The user interface is only partly translated. In the Bengali Wikipedia, the user interface parts must be translated to Bengali.

That's not intended to be the final outcome. To evaluate designs across teams English was used for the UI in the designs. I can make an example using English for content if that helps to review the idea in a more realistic representation (although we don't target English Wikipedia for this right now).

1. The "Review and fix" and the "Automatic translations are not always accurate" parts are small, gray, and too easy to miss.

I can iterate on that. Thanks for the feedback.

In T270714#7334353, @Amire80 wrote:

The working language in the design and development team is English, so it will be good to turn the prototypes around and use English as the target language. This will be the best thing for understanding the target user experience: the user interface will be readable to the whole team, and the imperfections of the machine translation will be constantly seen by everyone, too.

I created a new version that shows content and the UI in English. I used Portuguese to avoid previous concerns about low quality MT on less supported languages, and since the section existed in English Wikipedia I added two versions of the expanded content one with two paragraphs from MT and the other with two paragraphs of the English article for comparison:

@Amire80 please let me know if this is useful, and feel free to share new conclusions you got from the format you were proposing (or new adjustments that may be helpful).

Even with the scope of experimenting, there are a few issues we need to consider:

1. The problem statement "Provide readers access to knowledge that they could not access otherwise." should indicate that we are providing non-curated raw machine translation. Unavailability of detailed encyclopedic knowledge cannot immediately be solved raw machine translation as there is a probability of it being incorrect which can lead to misrepresentation of factual content. Machine translation for low resource languages is often known for low quality, factual errors, negation issues and may run into the above mentioned risk
2. " knowledge that they could not access otherwise." only implies that it cannot be accessed within Wikipedia. A content that is not present in Wikipedia may be accessible elsewhere on the web. We have seen in many of our research that Wikipedia is one among the many different ways to access knowledge, sometimes struggles to be appealing to newer generation of reader. Our value proposition guides the fact that we support exposing content that is goes through the editorial workflows followed by wiki contributors, even within the scope of an experiment or limited context.
3. We are living in a world of misinformation and fake news. One of the main characteristics of fake news is out of context(space and time) quotations. It is hard to prevent flow of misinformation from say, a screenshot of a machine translated content and present it as a screenshot of content from Wikipedia even if it is part of a limited experiment. Misinformation can spread because readers often avoid or do not realize the need for additional fact-checks. Extra banners suggesting this is machine translation is easy miss out in such contexts.
4. Wikipedia 's model of editing - the concept of wrong information by previous editors get quickly edited and improved by other editors - this has earned respect and it is one of the core principles of the integrity of the content. The urge to correct facts, improve articles are built on top of good faith on previous editors efforts. Their investment of time and effort to actually use of editing tools to improve articles must be respected. This investment of time and effort and their urge to fix things should not be exploited by showing raw machine translation and inviting to edit. I am afraid they are "edit baits" similar to click baits. It may not be respectful to their commitments to Wikipedia editing IMHO.

In T270714#7343805, @santhosh wrote:

Even with the scope of experimenting, there are a few issues we need to consider:

Thanks for the considerations, Santhosh. Some thoughts about these below.

1. The problem statement "Provide readers access to knowledge that they could not access otherwise." should indicate that we are providing non-curated raw machine translation. [...]Machine translation for low resource languages is often known for low quality, factual errors, negation issues and may run into the above mentioned risk

I agree that it is key to communicate very well that content is coming from machine translation, and consider issues with MT quality. If we expect serious issues such as factual errors and negations for certain languages, maybe we should consider a different set of languages for this, where such issues could be less frequent. Any ideas on how to evaluate for this are welcome. However, I still think that the Wikimedia communities are the best to say if this is useful for them or not based on how it works with their real articles (in a limited exposure to avoid disruptions).

1. " knowledge that they could not access otherwise." only implies that it cannot be accessed within Wikipedia.

Wikimedia projects are our focus. Our mission is to provide access to the sum of our human knowledge to anyone. In terms of equity, I don't think it is fair that Wikipedia provides for an English speaker detailed information about the Covid-19 vaccine while a Bulgarian only speaker or a Banjar-only speaker have access only to one or two paragraphs of content about the subject in the "equivalent" Wikipedia article. I think this goes against our mission. It may be hard to solve, but I think this is a problem worth exploring. The proposed approach is not perfect or a full solution, it has limitations and we need to evaluate it carefully but I don't think a valid answer from us is just to tell the Bulagrian or Banjar speakers to go elsewhere and look for reliable information on the subject in their language.

In addition, previous research shows that this language gap between large and medium-small wikis is demotivating for editors on the later. This means that the lack of content is also a barrier for these communities to grow.

Our value proposition guides the fact that we support exposing content that is goes through the editorial workflows followed by wiki contributors, even within the scope of an experiment or limited context.

One important consideration is that the machine translation text comes from the text created by Wikipedia editors. It is transformed and there can be aspects lost in translation, but we are reusing the contributions from editors. Also, this machine translation is not set in stone when showed, users can edit it as they incorporate it to the article (and are encouraged to do so) .

1. We are living in a world of misinformation and fake news. One of the main characteristics of fake news is out of context(space and time) quotations. It is hard to prevent flow of misinformation from say, a screenshot of a machine translated content and present it as a screenshot of content from Wikipedia even if it is part of a limited experiment. Misinformation can spread because readers often avoid or do not realize the need for additional fact-checks. Extra banners suggesting this is machine translation is easy miss out in such contexts.

I'm not sure I get this point. Currently people can make a full-page translation of any web page (including Wikipedia) in Google Translate or Google Chrome, and contents can be modified either on the browser or in the wiki themselves to make a screenshot. I don't see how exposing machine translated sections creates a more convenient vector for missinformation of this kind.

1. Wikipedia 's model of editing - the concept of wrong information by previous editors get quickly edited and improved by other editors - this has earned respect and it is one of the core principles of the integrity of the content. The urge to correct facts, improve articles are built on top of good faith on previous editors efforts. Their investment of time and effort to actually use of editing tools to improve articles must be respected. This investment of time and effort and their urge to fix things should not be exploited by showing raw machine translation and inviting to edit. I am afraid they are "edit baits" similar to click baits. It may not be respectful to their commitments to Wikipedia editing IMHO.

I don't see how showing a translation of content is disrespecting the editors. For those editors creating the original content, it is making their work more visible and reaching more people they could not reach because they were not able to contribute in such languages. There is a degradation of the message, but we are providing the mechanisms for other editors in such languages to work on it to get quickly edited and improved by other editors.

As you mentioned, building on top of others work is an important concept. Unfortunately for editors across different languages it is very hard to build on top of the work from others in a different language. Translation can actually help there, and machine translation scan be a convenient fist step that, while not perfect, showed in the past that can be useful. Tools such as Content Translation have been showing raw machine translation and inviting users to edit it, and many editors choose to use it, finding it convenient to reuse the work from other editors to easily create content in their language. We have heard about concerns on machine translation quality in some cases, but I'm not aware of issues about the process not fitting the wiki editing practices or the respect for editors investment in time and effort.