Page MenuHomePhabricator

Assessment to ensure that users can translate into/out of their app languages
Closed, InvalidPublic5 Story Points

Description

Assessment to ensure that users can translate into/out of their app languages

As a multilingual Wikipedia user, when I have proven myself as an editor of good standing in my primary language but not in other languages I speak, I want to translate title descriptions or image captions to other languages I speak, so that I can grow my contributions to Wikipedia.

Flow (Interactive flow chart | Download flow chart)

  1. Users either change the “From“ or “To“ language, as described in user story 1.
  2. On language change, a dialog appears that informs users about their first time translation attempt into the chosen language.
  3. Taping “Later“ resets the initial language selection.
  4. “Let’s go“ leads users to a language assessment that consists of three multiple choice questions. The exact questions have to be defined yet.
  5. a) If 3/3 questions have been answered correctly, the test has been completed successfully. Users are taken to the “Success“ view. A tap on “Get started“ takes them back to the Translate title description view. There, a new translation card based on the users language selection is presented.
  6. b) If the test hasn’t been completed successfully (1 or more errors), users are taken to the “Failed“ view. They can either tap “Skip“ to get back to the Translate title descriptions view or “Try again“ to take the test again. Infinite approaches are allowed.

Event Timeline

cmadeo created this task.Oct 22 2018, 6:35 PM
cmadeo added a comment.Dec 5 2018, 6:57 PM

Note: Update this ticket with mocks and descriptions that are based off of Dmitry's solution (multiple choice)

schoenbaechler renamed this task from Create a 'language CAPTCHA' as a way to ensure that users can translate into and out of their app languages to Assessment to ensure that users can translate into/out of their app languages.Dec 18 2018, 9:06 AM
schoenbaechler added a subscriber: Charlotte.EditedDec 19 2018, 4:35 PM

Adding @cmadeo’s comments from an earlier design draft:

Concerning 0101 on Zeplin:

​​I like the updated highlight here, Robin! Also something that came up with Dmitri yesterday was the question of if it would be possible to remove the ‘From’ and ‘To’ to increase space for the language name strings. He also mentioned that we’ll need to decide how the language names are displayed (eg. which language they should be displayed in).

In regards to user story 1:

​​These scenarios look good! One question that I have for the flow below though is what will happen if the user has more than 2 languages? Is the expectation that it will still default back to the Primary on switching langs? Even if I have German (primary), Italian and English and I start with German > English, when I switch the German to Italian, should the to switch to German? Or stay on English?

​​Ah! And sorry, to clarify the suggestion from Dmitry was to remove the To/From and use the two directional arrow icon that is used in Google translate instead

Adding @Charlotte’s comment from a call earlier:

It would be best if the language test is created/designed together with Pau, who’s the expert on the matter. Also, probably better if there are full sentences to translate and not only words that can easily be looked up with Google translate.

schoenbaechler updated the task description. (Show Details)

@cmadeo:

  • If users have added 3 or more languages to the app, it would not automatically adapt the To/From language. Automatic adaption would only be applied in the case of 2 languages. Specified it in the flow in the card’s description. Thanks for bringing that up.
  • Removed the labels from the select dropdown and updated the screens accordingly (CC @Dbrant)

Perfect, thanks for clarifying @schoenbaechler!
Would it be possible to also include the multi directional arrows in the translation mocks?

From Google translate:

schoenbaechler added a subscriber: Pginer-WMF.EditedDec 21 2018, 11:26 AM

I had a conversation with @Pginer-WMF yesterday about the difficulty of the language assessment. Based on Pau’s experience and after discussing the matter thoroughly, we came to the conclusion that we turn it into an exam and make it as accessible to users. Here’s our reasoning:

  • Given that users need to meet certain criteria to unlock title descriptions / image captions in the first place, we already have a quality assurance in place.
  • Secondly, users intentionally decide to use the translation feature in the edit action feed. They are taking the effort to navigate to it (also when prompted by a notification). This fact tells us that a particular user is willing and interested in contributing (quality) content.

In short, we think that the multiple choice questions, suggested in the designs are appropriate for this kind of test.

However, @Pginer-WMF suggested an alternative path to explore for incorporating an assessment like this. It’s more of an integrated approach where we would insert, e.g. 3 assessment translation cards (and indicate them as such), before the “real“ cards to translate are displayed. We would need to think about a mechanism to assure quality of the translations before these real cards are shown. These assessment cards would need to be simple as well. I find it a very natural and interesting approach, since users would already get familiar with the interface. I’ll explore this a bit after Christmas.

@schoenbaechler - The team has already discussed @Pginer-WMF's alternative suggestion. However, there is not at all an easy way to assess whether the translation given - if we allow the user to type in free text - is correct. We have no ability currently to have other users assess these "assessment translations". This is why for V1 we need something simple, like a multiple choice test. So by all means think about it, but we cannot build it in V1, and will need to go with the simpler multiple choice method initially.

schoenbaechler added a comment.EditedDec 21 2018, 2:49 PM

Thanks for the feedback @cmadeo. I added the language switch action, represented with the bidirectional arrow icon. Also added additional description on how it works - besides the obvious (swapping the language):

  • Tapping the language switch via bidirectional arrow icon in the main view also triggers the assessment dialog.

Thanks for clarifying @Charlotte and sounds good - I’ll think about Pau’s suggestion but let’s go with the simple version for now.

Charlotte updated the task description. (Show Details)Jan 7 2019, 4:17 PM

I created a flow chart for the translate title description assessment, to better understand and visualize the process.

Also, I’m working on an assessment that won’t require us to design a multiple choice test with made up questions. @Dbrant’s idea, which will reduce the implementation’s complexity quite a bit, in a nutshell:

  • Show a real article card that contains a correct title description/image caption in the users current translation language pair (e.g. German to French).
  • Users will only see title description/image caption in the original language (German) and have to choose the correct (French) translation for it out of three options.
  • The three multiple choice options consist of the correct Wikidata title description/image caption and two more random Wikidata title descriptions/image captions from other articles.
schoenbaechler added a comment.EditedJan 13 2019, 6:54 PM

Also, I’m working on an assessment that won’t require us to design a multiple choice test with made up questions. @Dbrant’s idea, which will reduce the implementation’s complexity quite a bit, in a nutshell:

  • Show a real article card that contains a correct title description/image caption in the users current translation language pair (e.g. German to French).
  • Users will only see title description/image caption in the original language (German) and have to choose the correct (French) translation for it out of three options.
  • The three multiple choice options consist of the correct Wikidata title description/image caption and two more random Wikidata title descriptions/image captions from other articles.

After spending time exploring this, I came up with the following:

  • Reused the existing concept/design as proposed in the card’s description, instead of using the interface for real article cards. This has the advantage that the exact same interface can be used for the “Translate title descriptions“ and “Translate image captions“ assessment.
  • The three multiple choice options consist of the correct Wikidata title description/image caption and two more random Wikidata title descriptions/image captions from other articles.
  • User flow stays the same as outlined (check out the flow chart)

Here’s how it’d look like:

Feel free to leave feedback here @Charlotte and @Dbrant. Also happy to talk about it in our 1:1 this week. Going to update Zeplin and the card’s description once we agreed on it.

Also CC’ing @cmadeo for design feedback 👋.

@schoenbaechler these look good to me! I'd just suggest creating a way to make both the string to be translated and the possible translations scrollable as some Wikidata descriptions and Image captions can be quite long.

Also, since users only need to pass this test once (and not for each task type) it should be okay to just use Title descriptions for this test.

Charlotte updated the task description. (Show Details)Jan 15 2019, 6:23 PM
Charlotte triaged this task as Normal priority.Jan 15 2019, 6:33 PM
Charlotte set the point value for this task to 5.
Mholloway added a comment.EditedJan 17 2019, 12:27 AM

Also, I’m working on an assessment that won’t require us to design a multiple choice test with made up questions. @Dbrant’s idea, which will reduce the implementation’s complexity quite a bit, in a nutshell:

  • Show a real article card that contains a correct title description/image caption in the users current translation language pair (e.g. German to French).
  • Users will only see title description/image caption in the original language (German) and have to choose the correct (French) translation for it out of three options.
  • The three multiple choice options consist of the correct Wikidata title description/image caption and two more random Wikidata title descriptions/image captions from other articles.

How do you anticipate gathering or generating a correct translation for an existing description or caption? If from Wikidata, it's not safe to assume that the existing descriptions for an entity in various languages are simply translations of one another.

For example, let's say we query the DB for entities with descriptions in both Arabic and Russian, and land on Q42, Douglas Adams. Here are the relevant descriptions per language:

LangValueEnglish translation (from Google Translate)
arكاتب إنجليزي فكاهيEnglish comic writer
ruанглийский писатель, драматург и сценарист, автор серии книг «Автостопом по галактике»English writer, playwright and screenwriter, author of the Hitchhiker's Guide to the Galaxy series of books
schoenbaechler added a comment.EditedJan 18 2019, 10:43 AM

Thanks for your feedback @Mholloway. How about adapting the multiple choice question language to reflect these Wikidata discrepancies?

Currently, the copy in the mocks reads:

Please choose the correct German translation for:

New copy:

Please choose the most appropriate German translation for:

How does that sound @Dbrant or @Charlotte? Maybe you have a better suggestion for it.

Charlotte removed schoenbaechler as the assignee of this task.Jan 19 2019, 12:11 AM
schoenbaechler updated the task description. (Show Details)

@schoenbaechler - Sorry Robin, didn't see your comment on the copy. I think "Please choose the most appropriate [language] translation..." sounds great.

Charlotte closed this task as Invalid.Mar 12 2019, 2:45 PM