Page MenuHomePhabricator

Content translation: Instrument publish_ attempt, failure, and success events
Closed, ResolvedPublic2 Estimated Story Points

Description

After the first iteration of analysing the translation funnel (T328913), the following publishing related events have been identified as priority to be instrumented:

  • publish_attempt: occurs when a user makes the final submission of the translation for publication
  • publish_failure: occurs when a user’s publication attempt fails due to a system error or limit
  • publish_success: occurs when a user's publication attempt is successful

Note:

Steps:

  • Instrumentation (Language team)
  • QA (Product Analytics)

Event Timeline

KCVelaga_WMF moved this task from Backlog to Priority Backlog on the Language-analytics board.
KCVelaga_WMF moved this task from Triage to Tracking on the Product-Analytics board.
KCVelaga_WMF renamed this task from [FY23-24 Q3 priority] Instrument publish_ attempt, failure, and success events to Content translation: Instrument publish_ attempt, failure, and success events.Jul 5 2024, 12:52 PM
KCVelaga_WMF lowered the priority of this task from High to Medium.
KCVelaga_WMF moved this task from Incoming to Engineering on the LPL Analytics board.
PWaigi-WMF raised the priority of this task from Medium to High.Aug 12 2024, 7:28 AM
abi_ set the point value for this task to 2.Aug 13 2024, 5:25 AM
SBisson moved this task from Prioritized to In-progress on the LPL Hypothesis board.
SBisson subscribed.

I'm a little unclear about the "publish_attempt" event. Do I understand correctly that it will be logged when the editor tries to publish their translation (when they click the publish button) and it will be immediately followed by a "publish_success" or "publish_failure" (when the publish API call returns)?

Do I understand correctly that it will be logged when the editor tries to publish their translation (when they click the publish button) and it will be immediately followed by a "publish_success" or "publish_failure" (when the publish API call returns)?

Yes, that's right. Here is the schema diagram: https://phab.wmfusercontent.org/file/data/dix6qzrnlsw4m5uqz5ei/PHID-FILE-rzfzwqmi2vu7guycykge/ContentTranslationEvent-flows-v5.png for reference. It will be triggered immediately after user clicks the button, followed by either success or failure event.

For more context (from documentation): Occurs when the user makes the final submission of the translation for publication. Does not occur when the user merely advances from the editing interface to the publication interface (for example, by selecting “done” in the mobile editor). The subtype publish_attempt_with_warnings is applied if there are unresolved warnings when the event occurs (including warnings manually dismissed by the user).

Hope that helps.

Also adding a note that, apart from the event itself, some other data points will need to be captured along with that (they are already present in the schema). That include:

  • human_modification_rate
  • human_modification_threshold
  • published_page_id
  • published_revision_id
  • translation_id
  • translation_type
  • translation_provider
  • translation_source_language
  • translation_source_section
  • translation_source_title
  • translation_target_exists
  • translation_target_language
  • translation_target_section
  • translation_target_title

For reference: Documentation/CX_event#interaction-related

[...] It will be triggered immediately after user clicks the button, followed by either success or failure event.

Thanks for clarifying. The "attempt" looks a little redundant. Is it serving a specific purpose or providing some information that "_success" or "_failure" don't have?

Is it serving a specific purpose or providing some information that "_success" or "_failure" don't have?

Not much generally, but provides additional information when a users forces to attempt publishing with warnings, for which the sub_type, publish_attempt_with_warnings should be recorded as well. The other way of capturing this information is, capturing it as sub_type of publish_success events, with something like publish_override_warnings.

publish_failure already has a sub-type publish_failure_due_to_warnings in the schema, so that is covered.

Note: Translation type should be an argument rather than being hard-coded as section. Please refer to T374905: Update translation_type in cx events to be an argument based on the type for details.

Change #1079007 had a related patch set uploaded (by Sbisson; author: Sbisson):

[mediawiki/extensions/ContentTranslation@master] Instrument publish_(attempt|success|failure)

https://gerrit.wikimedia.org/r/1079007

Change #1079007 merged by jenkins-bot:

[mediawiki/extensions/ContentTranslation@master] Instrument publish_(attempt|success|failure)

https://gerrit.wikimedia.org/r/1079007

Change #1085612 had a related patch set uploaded (by Nik Gkountas; author: Nik Gkountas):

[mediawiki/extensions/ContentTranslation@master] CX3 Build 0.2.0+20241101

https://gerrit.wikimedia.org/r/1085612

Change #1085612 merged by jenkins-bot:

[mediawiki/extensions/ContentTranslation@master] CX3 Build 0.2.0+20241101

https://gerrit.wikimedia.org/r/1085612

@KCVelaga_WMF This one is ready for you to verify. Thanks!

QA done! I couldn't trigger a publish_failure event, but from the data, I can verify that it is being logged.

Event counts so far:

event_typecount
publish_attempt874
publish_failure457
publish_success412