Page MenuHomePhabricator

Instrumentation: log recommendation difficulty level
Closed, ResolvedPublic2 Estimated Story Points

Description

Log recommendation difficulty level in a new field (translation_difficulty_level) of the content_translation_event schema for event type: dashboard_translation_start

Valid values for the translation_difficulty_level field are "stub", "easy", "medium", "hard", "unknown".

CX instrumentation schema where the new field is added

Derived Requirement

Update CX instrumentation to log the difficulty level of recommendations when a translation is started:

  • Add a new field translation_difficulty_level to the content\_translation\_event schema.
  • Apply this to the dashboard\_translation\_start event type.
  • The field must accept only the following values:
    • "stub"
    • "easy"
    • "medium"
    • "hard"
    • "unknown"
  • The value must reflect the assessed difficulty of the translation recommendation being started.
Test Steps

Test Case 1: Event Field Presence

  1. Start a translation from the dashboard.
  2. Capture the logged dashboard_translation_start event in the instrumentation.
  3. ✅❓❌⬜ AC1: The event contains the translation_difficulty_level field.

Test Case 2: Valid Value Logging

  1. Start translations for recommendations of different difficulty levels (stub, easy, medium, hard).
  2. Capture events for each.
  3. ✅❓❌⬜ AC2: The field value matches the expected classification of the recommendation.
  4. ✅❓❌⬜ AC3: All values are restricted to the allowed set: stub, easy, medium, hard, or unknown.

QA Results - TestWiki - Pending Connie

ACStatusDetails
1T399125#11211569
2T399125#11211569
3T399125#11211569

Event Timeline

SBisson updated the task description. (Show Details)
SBisson triaged this task as High priority.Jul 21 2025, 1:44 AM
ngkountas set the point value for this task to 2.Jul 30 2025, 1:34 PM

@cchen do you have recommendations about this task? Wondering where we could log the recommendation difficulty level and which events it applies to? The only other case I can find where we log more details about the recommendations is in event_source and it applies to various dashboard and editor events. See https://gitlab.wikimedia.org/repos/data-engineering/schemas-event-secondary/-/blob/master/jsonschema/analytics/mediawiki/content_translation_event/current.yaml#L140-149

@SBisson, I’d recommend adding a new column to the schema to log the recommendation difficulty level.
And it can apply to event types dashboard_translation_start.

@SBisson, I’d recommend adding a new column to the schema to log the recommendation difficulty level.
And it can apply to event types dashboard_translation_start.

Makes sense. Can you take care of adding the column and I'll populate the value in the code?

@SBisson, sure, to confirm the possible values for difficulty level will be easy, medium, and hard.

@cchen Right. And in some cases the level is unknown or unevaluated, should it be a different value or just omit the field in those cases?

@SBisson Hmmm, let's put 'unknown' then, in case there's any data issue and we need to check missing values and unknown values.

SBisson updated Other Assignee, added: cchen.
SBisson moved this task from Prioritized to In-progress on the LPL Hypothesis board.

@cchen I'm updating the recommendation API to also output difficulty level stub when the size is smaller than the "easy" threshold and unknown if it cannot be evaluated for whatever reason.

So all possible values are [ "unknown", "stub", "easy", "medium", "hard" ]

Please let me know when you can update the schema

Change #1183151 had a related patch set uploaded (by Sbisson; author: Sbisson):

[mediawiki/extensions/ContentTranslation@master] Include translation_difficulty_level in dashboard_translation_start

https://gerrit.wikimedia.org/r/1183151

Change #1183151 merged by jenkins-bot:

[mediawiki/extensions/ContentTranslation@master] Include translation_difficulty_level in dashboard_translation_start

https://gerrit.wikimedia.org/r/1183151

Change #1186648 had a related patch set uploaded (by Sbisson; author: Sbisson):

[mediawiki/extensions/ContentTranslation@master] CX3 Build 1.0.0+20250909

https://gerrit.wikimedia.org/r/1186648

Change #1186650 had a related patch set uploaded (by Sbisson; author: Sbisson):

[mediawiki/extensions/ContentTranslation@wmf/1.45.0-wmf.18] CX3 Build 1.0.0+20250909

https://gerrit.wikimedia.org/r/1186650

Change #1186648 merged by jenkins-bot:

[mediawiki/extensions/ContentTranslation@master] CX3 Build 1.0.0+20250909

https://gerrit.wikimedia.org/r/1186648

Change #1186650 merged by jenkins-bot:

[mediawiki/extensions/ContentTranslation@wmf/1.45.0-wmf.18] CX3 Build 1.0.0+20250909

https://gerrit.wikimedia.org/r/1186650

Mentioned in SAL (#wikimedia-operations) [2025-09-10T13:54:25Z] <kartik@deploy1003> sbisson, kartik: Backport for [[gerrit:1186650|CX3 Build 1.0.0+20250909 (T374886 T394998 T399122 T399125 T399133 T403730 T404045 T404093)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.

Mentioned in SAL (#wikimedia-operations) [2025-09-10T14:12:05Z] <kartik@deploy1003> Finished scap sync-world: Backport for [[gerrit:1186650|CX3 Build 1.0.0+20250909 (T374886 T394998 T399122 T399125 T399133 T403730 T404045 T404093)]] (duration: 24m 08s)

@SBisson Confirmed update CX instrumentation to log the difficulty level of recommendations when a translation is started, as seen below. I will move this to Sign-Off but Connie will be the one that Resolves it when she checks the events on the backend. Thanks for all your work!

@cchen Can you please validate and resolve? If you can check events when they end up in the backend, that would be great! Thanks!

Test Result - TestWiki

Status: ✅ PASS
Environment: TestWiki
OS: macOS Tahoe 26.0
Browser: Chrome 140
Device: MBA
Emulated Device: NA

Test Artifact(s):

https://test.wikipedia.org/w/index.php?title=Special:ContentTranslation&from=en&to=es&filter-type=automatic&filter-id=previous-edits&active-list=suggestions&page=Jebediah&section=Cover%20versions#/sx/sentence-selector
https://test.wikipedia.org/w/index.php?title=Special:ContentTranslation&from=en&to=hsb&filter-type=automatic&filter-id=previous-edits&active-list=suggestions&page=Duran%20Duran&section=Videos#/sx/sentence-selector
https://test.wikipedia.org/w/index.php?title=Special:ContentTranslation&from=en&to=es&filter-type=automatic&filter-id=previous-edits&active-list=suggestions&page=Aspic&section=Global%20variations%20of%20aspic#/sx/sentence-selector

Test Steps

Test Case 1: Event Field Presence

  1. Start a translation from the dashboard.
  2. Capture the logged dashboard_translation_start event in the instrumentation.
  3. AC1: The event contains the translation_difficulty_level field.
EasyMediumHard
2025-09-24_11-52-59.png (1×1 px, 412 KB)
2025-09-24_12-01-06.png (1×1 px, 396 KB)
2025-09-24_11-59-58.png (1×1 px, 448 KB)

Test Case 2: Valid Value Logging

  1. Start translations for recommendations of different difficulty levels (stub, easy, medium, hard).
  2. Capture events.
  3. AC2: The field value matches the expected classification of the recommendation.

See AC1

  1. AC3: All values are restricted to the allowed set: stub, easy, medium, hard, or unknown.

See AC1

I tested and confirmed the translation_difficulty_level logged in dashboard_translation_start events in mediawiki_content_translation_event table.