Page MenuHomePhabricator

Reorganize analytics for a Content & Section Translation unified experience
Closed, ResolvedPublic

Description

Currently Content and Section Translation are operating as almost separate tools: Content Translation is used for creating new articles on desktop and Section Translation is used on mobile, supporting expanding but also creating new articles.
The current dashboard and stats page reflects the old perspective.

The plan for the next developments is to provide a more unified experience where articles can be created and expanded in both desktop and mobile. In that scenario there will be a responsive tool with two editors (desktop and mobile) where both will be able to create and expand articles. Under this new context it makes sense to have a more integrated view of the analytics.

This ticket proposes to explore how to show the translation activity in a way can be filtered based on different criteria:

  • Activity: create or expand articles.
  • Platform: On desktop or on mobile.
  • Language: For all languages or a specific one.
  • Namespace: Content namespace or other (user drafts)

Timeline and dependencies

We may want to start thinking about this early. Even if some of the development to unify the translation experiecne will take time (not showing articles expanded on desktop until it is possible in the tool), some of the proposed tickets to expand the information of the current dashboard may get affected (T226170, T309603, T231321, T209868, T210138), and may be better to reorganize before expanding.

Dashboard duplication

Also this is an opportunity to decide about what to do with the current duplication of dashboards. The stats page was created before the team had support from the Analytics team. The page is unmaintained but provides some basic graphs visible to the community that show data in real time. The analytics dashboard is more recent and answers more recent questions but is not publicly visible and gets updated monthly.

Event Timeline

Pginer-WMF triaged this task as Medium priority.Jun 16 2022, 9:43 AM

@Pginer-WMF I've created an initial draft analytics dashboard for review, which shows one approach to showing translation activity based on a unified content and section translation experience. These updated analytics can be found under the Translation-Unified Experience tab.

See a summary of updates below and let me know if you have any questions. We can use this initial draft to iterate upon based on discussions.

Overview
The new dashboard shows all translations (combining translations tagged with either the section translation or contenttranslation tag) across all wikis. Current charts are provided to show the total (since deployed) translations, translations this month, year-over-year changes, translations by wiki, and by user edit count.

Filters
This data can then be filtered by the following dimensions. (Note each filter allows for multiple selections if you wish to view multiple items together (such as, a specific group of languages)).

  • Platform: Mobile Web and Other. In this case, 'Other' represents any non-mobile web or non-app edits. There is unfortunately no desktop category as there is no revision tag to distinguish desktop edits. However, in the case of translation edits, the majority of "Other" edits could be assumed to be desktop.
  • Page namespace: All page namespaces by name where translations have been completed
  • Creates_new_page: Did the translation create a new page? Need to confirm if this can be accurately used to distinguish between create and expand activity. See questions and notes below.
  • Language: Can be used to specify one or multiple languages. With no filter applied, the chart shows translations across all languages.

Questions:

  • Are there any additional charts or filters that would be useful?
    • I have a note to add a chart comparing published, published needing review, and deleted translations to be completed as part of T210138 and T209868.
  • Can the create_new_page filter correctly be used to indicate if the translation created a new article (creates new page = True) or expanded an article (creates new page = False)?
    • Note the creates_new_page dimension is based on whether the revision_parent_id = 0 in mediawiki_history.
    • I reviewed data collected in the dashboard to confirm if the numbers appear mostly as expected based on the current availability of the section and content translation tools. See notes from the review below:
      • Mobile Web: There have been about 10K mobile web translations recorded to date. These are all tagged with sectiontranslation as expected. About 51% of these translations were used to expand a new article (creates new page = False ) and 49% used to create a new article (creates new page = False ). Section translation tool can be used to create and expand so this seems correct.
      • Desktop: There have been 1.48M desktop edits (as indicated by platform = Other). The majority of these translations were used to create a new article (creates new page = True ); however, about 200K (14%) of these edits are recorded as expanding a translation. This is not expected as section translation is not currently available on desktop. This might be a case where non-desktop translations are being included in the platform = Other or a difference in how the create_new_page defines article creation compared to content translation . Will need to monitor to see if this number continue to grow.

@Pginer-WMF I've created an initial draft analytics dashboard for review, which shows one approach to showing translation activity based on a unified content and section translation experience. These updated analytics can be found under the Translation-Unified Experience tab.

Thanks Megan. This is great!
Overall, I think the options provide the ability to focus on the areas of interest that we were looking for. Being able to view the overall activity or focus on a specific area of interest.
One missing aspect is the time range selector. It is present in the Section Translaiton tab but not on the unified dashboard. The new unified dashboard seems to show data for the last 14 months (with a year-over-year graph in some cases which is really useful, by the way). However, it does not allow to observe trends for a longer period of time or focus on particular incidents. Examples:

  • I was comparing the recent translation activity between Catalan and Bengali, which is similar now, but I wanted to check how that compared prior to the enablement of mobile translation (Feb 20221) which Bengali Wikipedia has been active on.
  • If we want to look back at the data for the translation campaign that took place on Benglai Wikipedia between Feb-Mar 2022 in some months it will be our of the graph.

Regarding the more specific questions:

  • Are there any additional charts or filters that would be useful?
    • I have a note to add a chart comparing published, published needing review, and deleted translations to be completed as part of T210138 and T209868.

I think the current set covers the basics well for a first iteration. The following could be considered either to incorporate now (if they were trivial) or for future iterations):

  • Number of users and retention. How many different users participate in translation. Maybe this is worth considering combined with the analysis of retention (T226170)
  • Some of the data about entry point (T295757) or MT (T310773) reports. I don't know if it would be easy to integrate here, it would be preferred to have a separate dashboard or keep the report format to be run periodically. The aspects of those that could be more interesting to consider could be:
    • MT services used for the selected languages (interesting to correlate with deletions).
    • Percent of MT modifications for published translations (interesting to correlate with deletions)
    • Entry point events (interesting to correlate with published translations)
  • As you mentioned: published, published needing review, and deleted translations as part of T210138 and T209868.
  • Desktop: There have been 1.48M desktop edits (as indicated by platform = Other). The majority of these translations were used to create a new article (creates new page = True ); however, about 200K (14%) of these edits are recorded as expanding a translation. This is not expected as section translation is not currently available on desktop. This might be a case where non-desktop translations are being included in the platform = Other or a difference in how the create_new_page defines article creation compared to content translation . Will need to monitor to see if this number continue to grow.

On Dektop expanding a section is possible but only in recent months and when knowing the URL for it since it has not been exposed in the dashboard. Thus, I think there are two kinds of data mixed that we may want to separate:

  • Expanding by translating a new section. These use the new section translation editor on desktop and are marked with the "sectiontranslation" tag in addition to the "contenttranslaiton" one. There will be very few of them, just from internal testing.
  • Article overwrites. A translation is published when there is already an existing article . This can be overwriting a previous article by a different author or publishing a translation multiple times by the same author as a way to continue translating more content. These will have the "contenttranslation" tag but not the "sectiontranslation" one.

@Pginer-WMF - Thanks for the review and feedback. I've updated the dashboard with several of the suggested changes. See a summary of updates below.

Changes Incorporated in this Iteration

  • Time Range Selector: This allows the dashboard to be filtered to a specific time frame. Note that the default is currently set to show last 2 years, when you first open the dashboard.
  • Total translations since deployment. I've added a chart to indicate the total number of translations (including both content and section) since deployment. This chart is static, meaning that it will not change based on any filter inputs provided by the user. Please let me know if there are certain filters that would be useful to still apply to this chart as well and I can adjust them. Note: All other charts on the dashboard will update based on the filter inputs.
  • Translation Activity Filter: I've added a new filter to help decipher between the three types of translation activity listed below. I've run a few checks to confirm that each of these activities captures the correct translations. Numbers appear correct to me but let me know if anything appears unexpected.
    • expand a translation.
    • create a new translation
    • overwrite an article.

Changes Recommended for Next Iteration
The other additions discussed would require querying and importing new data into Superset, which would be a larger effort and would likely be suited under separate tasks. See notes below regarding how we might track and add these other metrics to the dashboard:

  • Number of users and retention. How many different users participate in translation. Maybe this is worth considering combined with the analysis of retention (T226170)

I've added a note to the description of T226170 to add the resulting retention data to the Content Translation key metrics dashboard once the analysis is complete.

  • Some of the data about entry point (T295757) or MT (T310773) reports. I don't know if it would be easy to integrate here, it would be preferred to have a separate dashboard or keep the report format to be run periodically. The aspects of those that could be more interesting to consider could be:
    • MT services used for the selected languages (interesting to correlate with deletions).
      • Percent of MT modifications for published translations (interesting to correlate with deletions)
      • Entry point events (interesting to correlate with published translations)

It's possible to integrate data from both of these report into the dashboard but the process would be complicated, as they require querying and importing datasets not already available within Superset. There is currently an open task (T287306) to create an ETL job that would periodically calculate these metrics and save them to the Data Lake. Once resolved, this would allow us to more easily access these new metrics without timeout errors.
Pending resolution of that task, it would be easier to rerun the reports periodically and provide a summary of the metrics of interest.

  • As you mentioned: published, published needing review, and deleted translations as part of T210138 and T209868.

Will be completed as part of T210138 and T209868.

Thanks for the great work on this, Megan.
The dashboard looks good to me and I agree most of the additions can be considered as part of separate tickets (or periodically updated reports).

I just have a couple of additional comments:

  • For the "Total translations since deployment" we could either (a) apply all filters except the time range or (b) have a set of separate counters for: total, new articles mobile, new articles desktop, sections mobile and sections desktop. Whichever is easier to support.
  • The time range selector has no default value for the "end" field. However, when this is left empty, the "Monthly translations year over year" breaks because it needs such value. Having "Today" as a default value for the end period would avoid this issue. If it is not possible to have a predefined value for this, we can totally live with this (just need to remember to add it manually when interested in such graph)

@Pginer-WMF - See updates to dashboard below.

For the "Total translations since deployment" we could either (a) apply all filters except the time range or (b) have a set of separate counters for: total, new articles mobile, new articles desktop, sections mobile and sections desktop. Whichever is easier to support.

I went with option (a) and applied all filters except the time range for the "Total translations since deployment". Now, you can use the other filters on the dashboard to identify the total translations since deployment for any selected combination of platform, activity, language, and namespace. Note there is also a "Total translations for selected time range" count, which can be used if you're interested in the number of translations within a specific time range.

The time range selector has no default value for the "end" field. H

Thank you for catching this! I've updated the time range selector to have a default "end" time of "Today". This should help prevent users from leaving this field empty when adjusting the time range.

Perfect. Thanks @MNeisler !
Option (a) seems great. One minor adjustment to consider in terms of layout. Would it be possible to have the "Total translations since deployment" number next to the "Total translations for selected time range" in the dashboard. In that case we don't need to have a separate section at the beginning, connection with the filters may be more clear, and comparison with the numbers more direct.
Given that this is for internal use, consider this only if it were trivial to apply.

Thanks in advance!

Sure, no problem @Pginer-WMF. I've adjusted the layout to have the "Total translations since deployment" number next to the "Total translations for selected time range" in the dashboard.

Perfect. The dashboard looks great. Thanks for all the adjustments!

(No further nitpicks from my side)