Page MenuHomePhabricator

Content translation: Instrument editor_open and editor_close events
Closed, ResolvedPublic

Description

After the first iteration of analysing the translation funnel (T328913), the following editing screen related events have been identified as priority to be instrumented:

Note: No schema changes required

Steps

  • Instrumentation
  • QA

Event Timeline

KCVelaga_WMF renamed this task from [Q3 priority] Instrument editor_open and editor_close events to [FY23-24 Q3 priority] Instrument editor_open and editor_close events.
KCVelaga_WMF triaged this task as High priority.
KCVelaga_WMF renamed this task from [FY23-24 Q3 priority] Instrument editor_open and editor_close events to Content translation: Instrument editor_open and editor_close events.Jul 5 2024, 12:52 PM
KCVelaga_WMF lowered the priority of this task from High to Medium.
KCVelaga_WMF moved this task from Incoming to Engineering on the LPL Analytics board.
PWaigi-WMF raised the priority of this task from Medium to High.Sep 3 2024, 10:00 AM

@KCVelaga_WMF The definition of the editor_open event and the possible event sources in the schema documentation, seems to be based on the Content Translation workflow.

In Section Translation we have the "Pick a sentence" step (see screenshot 1), and then we have the "Edit a sentence" step (screenshot 2). Usually, we refer to the "Edit a sentence" step as "editor", since this is the step where the actual editing happens. However, for the context of this task, I believe that the "editor_open" and "editor_close" events should be logged when user enters/leaves the "Pick a sentence" step. What do you think?

Pick a sentence step:

Screenshot 2024-09-11 at 20.11.03.png (1×796 px, 220 KB)

Edit a sentence:

Screenshot 2024-09-11 at 20.11.13.png (1×796 px, 49 KB)

Additionally, regarding the possible event sources, I believe that it doesn't make sense to just instrument the content_language_selector, maybe we want to add all other entrypoints as possible sources, too. Additionally, I don't think there is a way to detect whether a user is coming from a deletion notification, in SX/Unified Dashboard.

@Pginer-WMF you may also have some thoughts about the above.

@ngkountas

The definition of the editor_open event and the possible event sources in the schema documentation, seems to be based on the Content Translation workflow. In Section Translation we have the "Pick a sentence" step (see screenshot 1), and then we have the "Edit a sentence" step (screenshot 2). Usually, we refer to the "Edit a sentence" step as "editor", since this is the step where the actual editing happens. However, for the context of this task, I believe that the "editor_open" and "editor_close" events should be logged when user enters/leaves the "Pick a sentence" step. What do you think?

Good point, and that sounds good (about editor/open closes for "Pick a sentence" step). In that case, it will be helpful if we can log another event when users proceeds from the translation start screen to the screen where they can edit. It will be a new event in that case. I will include it as part of T373785.

Additionally, regarding the possible event sources, I believe that it doesn't make sense to just instrument the content_language_selector, maybe we want to add all other entrypoints as possible sources, too. Additionally, I don't think there is a way to detect whether a user is coming from a deletion notification, in SX/Unified Dashboard.

I think we can hold off on sources for editor_open and editor_close events in this case. Because the sources in that case would actually be for user reaching the step of editing screen (rather the "Pick a sentence" step). I think content_language_selector should be removed (it was from the old documentation, I will update that). As it is a source for the translation start screen, rather than the editing screen.


An unrelated question to the task: when a user clicks on starting a new translation (i.e. translating an article) and the further steps, the translation_type is being logged as section, but not as article. Shouldn't translation type section be limited to events where users is expanding a sections of existing article?

If you ask me, none of the sources makes a lot of sense. content_language_selector, direct, and notification_deletion are all possible sources for the dashboard_open event, while dashboard_translation_continue and dashboard_translation_start are separate events, which means we can easily check if the user is coming to the editor ("Pick a sentence" step) by starting or continuing a translation, by checking the previous logged event (should be one of those two).


Regarding the unrelated question: You are right. Currently, the value for translation_type is hardcoded to section, but it should be given as an argument, depending the case. Do you think we need to file a new ticket for this?

If you ask me, none of the sources makes a lot of sense. content_language_selector, direct, and notification_deletion are all possible sources for the dashboard_open event, while dashboard_translation_continue and dashboard_translation_start are separate events, which means we can easily check if the user is coming to the editor ("Pick a sentence" step) by starting or continuing a translation, by checking the previous logged event (should be one of those two).

Sounds good to me. I agree that it is kind of redundant to capture them as source.


Currently, the value for translation_type is hardcoded to section, but it should be given as an argument, depending the case. Do you think we need to file a new ticket for this?

Yes, I will file a separate ticket for that.

Change #1074399 had a related patch set uploaded (by Nik Gkountas; author: Nik Gkountas):

[mediawiki/extensions/ContentTranslation@master] SX: Add instrumentation for "editor_open" event

https://gerrit.wikimedia.org/r/1074399

Change #1074401 had a related patch set uploaded (by Nik Gkountas; author: Nik Gkountas):

[mediawiki/extensions/ContentTranslation@master] SX: Add instrumentation for "editor_close" and "editor_close_warn" events

https://gerrit.wikimedia.org/r/1074401

@KCVelaga_WMF while testing the patch for the editor_open event, I noticed that many events were failing validation because the "translation_provider" field doesn't have "MinT" in its allowed values. Could you add it and publish a new version of the schema? Thanks

Change #1074399 merged by jenkins-bot:

[mediawiki/extensions/ContentTranslation@master] SX: Add instrumentation for "editor_open" event

https://gerrit.wikimedia.org/r/1074399

Change #1076902 had a related patch set uploaded (by KCVelaga; author: KCVelaga):

[schemas/event/secondary@master] Add MinT translation provider

https://gerrit.wikimedia.org/r/1076902

@SBisson Ah, should have added it a while back. Created a patch for the same. Once merged, the latest schema will be /analytics/mediawiki/content_translation_event/1.5.0

Change #1076902 merged by jenkins-bot:

[schemas/event/secondary@master] Add MinT translation provider

https://gerrit.wikimedia.org/r/1076902

@KCVelaga_WMF I approved the above patch to add the MinT translation provider, but since we are here, let's fix all the translation providers. According to the CX documentation the following MT providers are supported:

  • MinT
  • Apertium
  • LingoCloud
  • Google Translate
  • Yandex
  • Elia

This means that the allowed values for translation_provider field should be:

  • apertium
  • blank
  • elia
  • google
  • lingocloud
  • mint
  • source
  • yandex

In practice, we should remove matxin (T285199), opusmt (T337657) and youdao (T329137) MT providers, and also add Elia in the list.

Change #1076988 had a related patch set uploaded (by KCVelaga; author: KCVelaga):

[schemas/event/secondary@master] Update translation provider enum values

https://gerrit.wikimedia.org/r/1076988

Change #1076993 had a related patch set uploaded (by KCVelaga; author: KCVelaga):

[schemas/event/secondary@master] Add elia to translation providers

https://gerrit.wikimedia.org/r/1076993

Change #1076988 abandoned by KCVelaga:

[schemas/event/secondary@master] Update translation provider enum values

https://gerrit.wikimedia.org/r/1076988

Change #1076996 had a related patch set uploaded (by KCVelaga; author: KCVelaga):

[schemas/event/secondary@master] Add elia translation provider

https://gerrit.wikimedia.org/r/1076996

Change #1076996 merged by jenkins-bot:

[schemas/event/secondary@master] Add elia translation provider

https://gerrit.wikimedia.org/r/1076996

Change #1074401 merged by jenkins-bot:

[mediawiki/extensions/ContentTranslation@master] SX: Add instrumentation for "editor_close" and "editor_close_warn" events

https://gerrit.wikimedia.org/r/1074401

Change #1079550 had a related patch set uploaded (by Nik Gkountas; author: Nik Gkountas):

[mediawiki/extensions/ContentTranslation@master] CX3 Build 0.2.0+20241011

https://gerrit.wikimedia.org/r/1079550

Change #1079550 merged by jenkins-bot:

[mediawiki/extensions/ContentTranslation@master] CX3 Build 0.2.0+20241018

https://gerrit.wikimedia.org/r/1079550

Change #1076993 abandoned by KCVelaga:

[schemas/event/secondary@master] Add elia to translation providers

https://gerrit.wikimedia.org/r/1076993

I did the QA - editor_open, editor_close, and editor_close_warn are being logged.

And from the data, during the last two weeks, here is the quick of overview of the events logged.

eventcount
editor_open2837
editor_close631
editor_close_warn17