Page MenuHomePhabricator

Measure success and failure rates for newcomers using Content Translation
Open, NormalPublic

Description

As part of the success metrics identified for Content Translation version 2, we want the success rate for new editors using the tool to increase.

Measurement

We want to measure the percentage of translations by (new) editors that are completed successfully.

  • We want to have separate measurements for (a) "new editors", and (b) "existing editors" as a reference.
  • We want to have separate measurements for activity from (a) all versions, and (b) version 2 of the tool. That will allow to understand the overall impact as version 2 becomes the default but also to "zoom in" to focus only on version 2 (where the improvements are made) when needed.
  • Success levels: We want to capture the translations that resulted in a published article (success), those that were not published because an error prevented from publishing, and those where users just abandoned the process and leave the translation incomplete for the given time period (i.e. the user started the translation but didn't pressed "publish").
  • New editors are defined as users that created their account during the last 6 months (the initial criteria user for new editor experiences research).
  • Time period: We want to group the results in 30 day periods. Analysing the translations that were started in a given month. We may need to consider a delay to give enough time to be sure that "abandoned" translations are not misclassified.

Representation

An example representation is shown below, where the percentage of successfully published translations increases over time, as errors and abandoned translations decrease. The graph shows, regardless the number of new users or the translations they make, how likely they are to succeed or face problems.

Event Timeline

Restricted Application added a project: ContentTranslation. · View Herald TranscriptMay 14 2018, 11:03 AM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
Pginer-WMF updated the task description. (Show Details)May 14 2018, 11:05 AM
Pginer-WMF triaged this task as Normal priority.May 14 2018, 5:32 PM
Pginer-WMF moved this task from Needs Triage to CX2 on the ContentTranslation board.
Pginer-WMF updated the task description. (Show Details)May 23 2018, 8:10 AM
Pginer-WMF updated the task description. (Show Details)May 29 2018, 3:12 PM
Pginer-WMF updated the task description. (Show Details)
Pginer-WMF updated the task description. (Show Details)May 29 2018, 4:54 PM
Vvjjkkii renamed this task from Measure success and failure rates for newcomers using Content Translation to g0caaaaaaa.Jul 1 2018, 1:10 AM
Vvjjkkii raised the priority of this task from Normal to High.
Vvjjkkii updated the task description. (Show Details)
Vvjjkkii removed a subscriber: Aklapper.
CommunityTechBot renamed this task from g0caaaaaaa to Measure success and failure rates for newcomers using Content Translation.Jul 2 2018, 5:34 AM
CommunityTechBot lowered the priority of this task from High to Normal.
CommunityTechBot updated the task description. (Show Details)
CommunityTechBot added a subscriber: Aklapper.
Pginer-WMF updated the task description. (Show Details)Jul 5 2018, 4:41 PM
Neil_P._Quinn_WMF claimed this task.EditedJul 11 2018, 4:02 PM

A user starting multiple translations, should be counted only once.

@Pginer-WMF, I don't understand this part. If a user started multiple translations, why would we omit all but one from the calculation? After all, (new) users returning is a big part of success. Also, there's the issue of which of the user's translations we should count: the first? the last?

Maybe this is mistaken carryover from T194641?

A user starting multiple translations, should be counted only once.

@Pginer-WMF, I don't understand this part. If a user started multiple translations, why would we omit all but one from the calculation? After all, (new) users returning is a big part of success. Also, there's the issue of which of the user's translations we should count: the first? the last?
Maybe this is mistaken carryover from T194641?

I guess I was worried about us reading the graph and saying "75% of the users were able to complete their translations successfully" and those not being 75% of the users because of users making multiple translations. But you are right, there is no way to summarize all the experiences of a user into a single one if those were different. I guess that we can count each translation attempt and talk about "75% of the times users are able to complete their translations successfully".

I'll adjust the description to avoid confusions.

Pginer-WMF updated the task description. (Show Details)Jul 11 2018, 4:23 PM

quick drop-by to say that dashiki has support for these stacked bars (though no dashboards are currently using them).

kzimmerman removed chelsyx as the assignee of this task.Jul 9 2019, 9:20 PM
kzimmerman added subscribers: chelsyx, kzimmerman.

Unassigning @chelsyx because she's moving on from the Foundation.