Page MenuHomePhabricator

MinT for Readers: Implement instrumentation for key events
Closed, ResolvedPublic4 Estimated Story Points

Description

As part of the work on MinT for Wikipedia Readers MVP (T359072), an schema for instrumentation has been defined using the new data platform. This ticket proposes to implement the instrumentation for a set of key events. You can check the documentation for more details on how to use the new platform in the implementation.

Form the designed schema (T341185) the key events are the following:

  • session initiation
  • users searches for a topic
  • users selects an article
  • user clicks to view to automatic translation
  • user clicks to view human generated content
  • user closes the automatic translation view

Derive Requirement

Implement the instrumentation for key user events related to the MinT MVP for Wikipedia Readers using the new data platform. The key events to be tracked are:

Session initiation
User searches for a topic
User selects an article
User clicks to view automatic translation
User clicks to view human-generated content
User closes the automatic translation view
All events should log the required properties to ensure proper tracking and analytics.

BDD

Feature: MinT MVP for Wikipedia Readers instrumentation

Scenario: Session initiation
  Given a user visits the Wikipedia site
  When the user lands on the automatic translation page
  Then the session initiation event should be logged

Scenario: User searches for a topic
  Given the user is on the automatic translation page
  When the user searches for a topic
  Then the "search_topic" event should be logged

Scenario: User selects an article
  Given the user has searched for a topic
  When the user selects an article from the search results
  Then the "select_article" event should be logged

Scenario: User clicks to view automatic translation
  Given the user has selected an article
  When the user clicks to view the automatic translation
  Then the "view_automatic_translation" event should be logged

Scenario: User clicks to view human-generated content
  Given the user is on the automatic translation page
  When the user clicks to view the human-generated content
  Then the "view_human_content" event should be logged

Test Steps

Pre-requisite:

-Ensure debug logs are enabled by executing the following code in the browser developer console:
javascript
Copy code:

mw.loader.using( 'mediawiki.api' ).then( function () {
  new mw.Api().saveOption( 'eventlogging-display-web', '1' );
});

Test Case 1: Session initiation

Visit a Wikipedia site (e.g., https://es.wikipedia.org).
Open the Automatic Translation special page.
Expected Result:
✅❓❌⬜ AC1: Confirm that a "session initiation" event is logged in the console, containing the relevant session properties.

Test Case 2: User searches for a topic

On the Automatic Translation page, search for a topic.
Expected Result:
✅❓❌⬜ AC2:Confirm that a "search_topic" event is logged in the console, including properties like action: "search", action_source: "topic_search".

Test Case 3: User selects an article
After searching for a topic, select an article from the search results.
Expected Result:
✅❓❌⬜ AC3:Confirm that a "select_article" event is logged, with properties like action: "click", action_source: "article_selection".

Test Case 4: User clicks to view automatic translation
After selecting an article, click to view the automatic translation.
Expected Result:
✅❓❌⬜ AC4:Confirm that a "view_automatic_translation" event is logged, including properties such as action: "click", action_source: "automatic_translation_card".

Test Case 5: User clicks to view human-generated content
Click on the option to view human-generated content (if available).
Expected Result:
✅❓❌⬜ AC5:Confirm that a "view_human_content" event is logged, with properties like action: "click", action_source: "human_content".

QA Results - ES Automatic Translation PROD

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Change #1048400 had a related patch set uploaded (by Nik Gkountas; author: Nik Gkountas):

[mediawiki/extensions/ContentTranslation@master] AX instrumentation: Update stream name

https://gerrit.wikimedia.org/r/1048400

Thank you @KCVelaga_WMF! I just submitted the patch for updating the stream name. Do you need me to review your patch, too?

@ngkountas Yes, I have add you and engineers from the Metrics Platform team as reviewers for that.

Change #1048400 merged by jenkins-bot:

[mediawiki/extensions/ContentTranslation@master] AX instrumentation: Update stream name

https://gerrit.wikimedia.org/r/1048400

Based on the QA comments (copied below) there are a couple of aspects to polish for the instrumentation to work as expected. Moving the ticket back to in development:

  • Approximately ~560 events had validation errors till now. The error is the same for all of them: '.performer' should NOT have additional properties
    • When I checked the raw events, event were using a different schema version than the rest i.e. /analytics/product_metrics/web/base/1.0.0 instead of /analytics/product_metrics/web/base/1.2.0
  • Also, it seems like all the errored events were only users clicking the automatic translation card. If a user selected the automatic translation of an article, there should be a preceding event of article selection (which is part of the key events), and that seems to be missing. Can you please check if this being logged and also the human translation selection?

These validation errors are not caused by our implementation, but rather by a bug in the mw.eventLog.submitClick method of the Metrics Platform repository. All other events, that are not click events, (e.g. search, session_init etc) are properly logged, as far as I am concerned. A pull request has been submitted to fix the issue with the above method, and according to the plan it will be backported on Monday.

I'm moving this task to "Waiting for deployment" column. @KCVelaga_WMF do you think you could QA this task once the PR is merged and deployed?

I'm moving this task to "Waiting for deployment" column. @KCVelaga_WMF do you think you could QA this task once the PR is merged and deployed?

The change has been deployed. I'm monitoring https://logstash.wikimedia.org/goto/504f7423ac1d4643e285e8cf4b80cb33.

Thank you @phuedx. I don't see any more validation errors related the schema version. The data also has click events for automatic and human translation selections.

@ngkountas Can you double check if the preceding event to user selecting an article from search before selecting automatic translation or human translation is being logged or not? (3rd event in the list of keys events)

I can confirm that click events for user selecting an article are being logged. We have ~30+ events so far.

However, we have ~1000+ events for users clicking the auto_translation_card, so I am wondering about the possibility if users can go to clicking the automatic translation card, without selecting an article.

However, we have ~1000+ events for users clicking the auto_translation_card, so I am wondering about the possibility if users can go to clicking the automatic translation card, without selecting an article.

Not sure about the instrumentation implementation details. But in terms of the user workflow there are multiple possible paths.
The general workflow consists of the following steps Home -> Search -> Confirm -> Translation View
Users can directly access any of them through the url. Entry points direct them to the Confirm step directly where the article and language pairs have been selected based on the context where the user comes form.

QA Results

The following events have been verified to be properly logged:

session_init (session initiation):

search (users searches for a topic):

"search_result" click (users selects an article):

"auto_translation_card" click (user clicks to view to automatic translation):

"human_translation_card" click (user clicks to view human generated content):

Missing piece: The only current unsupported event for this task is the "close" click event (user closes the automatic translation view). Moving this task back to priority log, until this is also fixed.

PWaigi-WMF renamed this task from MinT MVP: Implement instrumentation for key events to MinT for Readers: Implement instrumentation for key events .Aug 20 2024, 8:57 AM

Change #1064047 had a related patch set uploaded (by Nik Gkountas; author: Nik Gkountas):

[mediawiki/extensions/ContentTranslation@master] MinT for Readers: Log "view" event after content has been loaded

https://gerrit.wikimedia.org/r/1064047

Change #1064048 had a related patch set uploaded (by Nik Gkountas; author: Nik Gkountas):

[mediawiki/extensions/ContentTranslation@master] MinT for Readers: Instrument "close automatic translation" event

https://gerrit.wikimedia.org/r/1064048

Change #1064047 merged by jenkins-bot:

[mediawiki/extensions/ContentTranslation@master] MinT for Readers: Log "view" event after content has been loaded

https://gerrit.wikimedia.org/r/1064047

Change #1064048 merged by jenkins-bot:

[mediawiki/extensions/ContentTranslation@master] MinT for Readers: Instrument "close automatic translation" event

https://gerrit.wikimedia.org/r/1064048

@ngkountas Is this QA testable and if so, can you please provide me the QA steps for me? Thanks!

@GMikesell-WMF yes, this task is QA testable. This is a task about instrumentation, so the easiest way to reproduce it, is to enable debug logs for each logged event inside your browser developer console, and also as popups inside your screen.

Step 1:
To enable these logs, visit a production Wikipedia site (e.g. https://es.wikipedia.org), open your browser developer console, and execute this small piece of code:

mw.loader.using( 'mediawiki.api' ).then( function () {
    new mw.Api().saveOption( 'eventlogging-display-web', '1' );
} );

After refreshing the page, the debug logs will be enabled, and you'll see a popup message inside your screen, and a console message (with a JSON object) for every instrumentation event logged in the site.

Step 2:
Visit the Automatic Translation special page -aka MinT MVP for Readers - in any wiki (preferably not English wiki). Search for any article to get the automatic translation, and continue all the way until the final screen, where the user can read the automatic translation of the article. You can see my screencasts above, for some examples of how this tool works. For instance, if you check the screencast for the test scenario "auto_translation_card" click (user clicks to view to automatic translation) above, you can see the "View Automatic Translation" page, where the user can read the automatic translation of the article.

Step 3:
Inside this "View Automatic Translation" screen, click on the close button (next to the "Automatic Translation" header text). A new instrumentation event should be logged. You can check your browser dev console to find the logged event object. That object should contain an action property equal to "click", an action_source property equal to "automatic_translation_header" and an action_subtype property equal to "close".

If this is the case, it means that the last remaining event is also instrumented properly and this task can be closed as done.

@ngkountas Thanks for the QA steps! As seen in AC5 gif, when I clicked on the human article, it did not generate. The rest logs and looks fine though.

Test Result - ES AutomaticTranslation PROD

Status: ❌ FAIL AC5
Environment: ES AutomaticTranslation PROD
OS: macOS Sonoma 15.0
Browser: Chrome 129
Device: MBA
Emulated Device: NA

Test Artifact(s):
https://es.wikipedia.org/wiki/Especial:AutomaticTranslation

Test Steps

Pre-requisite:

-Ensure debug logs are enabled by executing the following code in the browser developer console:
javascript
Copy code:

mw.loader.using( 'mediawiki.api' ).then( function () {
  new mw.Api().saveOption( 'eventlogging-display-web', '1' );
});

Test Case 1: Session initiation

  1. Visit a Wikipedia site (e.g., https://es.wikipedia.org).
  2. Open the Automatic Translation special page.
  3. ✅ AC1: Confirm that a "session initiation" event is logged in the console, containing the relevant session properties.

session_init (session initiation):

2024-09-27_13-15-43.mp4.gif (1×1 px, 1 MB)

Test Case 2: User searches for a topic

  1. On the Automatic Translation page, search for a topic.
  2. ✅ AC2:Confirm that a "search_topic" event is logged in the console, including properties like action: "search", action_source: "topic_search".

search (users searches for a topic):

2024-09-27_13-16-21.mp4.gif (1×1 px, 1 MB)

Test Case 3: User selects an article

  1. After searching for a topic, select an article from the search results.
  2. ✅AC3:Confirm that a "select_article" event is logged, with properties like action: "click", action_source: "article_selection".

"search_result" click (users selects an article):

2024-09-27_13-16-47.mp4.gif (1×1 px, 1 MB)

Test Case 4: User clicks to view automatic translation

  1. After selecting an article, click to view the automatic translation.
  2. ✅ AC4:Confirm that a "view_automatic_translation" event is logged, including properties such as action: "click", action_source: "automatic_translation_card".

"auto_translation_card" click (user clicks to view to automatic translation):

2024-09-27_13-17-21.mp4.gif (1×1 px, 2 MB)

Test Case 5: User clicks to view human-generated content

  1. Click on the option to view human-generated content (if available).
  2. ❌ AC5:Confirm that a "view_human_content" event is logged, with properties like action: "click", action_source: "human_content".

Human translation card did not display as seen in the gif. Is it because it opens a new tab?

"human_translation_card" click (user clicks to view human generated content):

2024-09-27_13-17-56.mp4.gif (1×1 px, 2 MB)

Change #1076739 had a related patch set uploaded (by Nik Gkountas; author: Nik Gkountas):

[mediawiki/extensions/ContentTranslation@master] AX MVP: Instrument "open target article" inside view translation

https://gerrit.wikimedia.org/r/1076739

Change #1076739 merged by jenkins-bot:

[mediawiki/extensions/ContentTranslation@master] AX MVP: Instrument "open target article" inside view translation

https://gerrit.wikimedia.org/r/1076739

Nikerabbit raised the priority of this task from Medium to High.Feb 17 2025, 8:10 AM
Nikerabbit set the point value for this task to 4.

QA results

session initiation and search

✅ all good

Screenshot from 2025-02-25 11-15-40.png (1×1 px, 231 KB)

@ngkountas I have a question here, right now, we don't have any entry points enabled so the only way to direct open the special page and init event being logged fine. When we have the other entry points, like article footer, will the action_source be captured accordingly? For the experiment, I think we have article footer, languages selector and search.

users selects an article

✅ all good

Screenshot from 2025-02-25 11-16-14.png (1×1 px, 295 KB)

user clicks to view to automatic translation and view event

✅ all good

Screenshot from 2025-02-25 11-16-34.png (1×1 px, 365 KB)

Screenshot from 2025-02-25 11-17-08.png (1×1 px, 383 KB)

user closes the automatic translation view

✅ all good

Screenshot from 2025-02-25 11-19-38.png (1×1 px, 191 KB)

user clicks to view human generated content

It seems like this event is not being logged. I tried at multiple instances.

Nikerabbit updated Other Assignee, added: ngkountas.

Change #1125375 had a related patch set uploaded (by Abijeet Patro; author: Abijeet Patro):

[mediawiki/extensions/ContentTranslation@master] AX Instrumentation: Fix human_translation_card event

https://gerrit.wikimedia.org/r/1125375

Submitted a patch to fix the issue with human_translation_card:

image.png (1×2 px, 214 KB)

Change #1125375 merged by jenkins-bot:

[mediawiki/extensions/ContentTranslation@master] AX Instrumentation: Fix human_translation_card event

https://gerrit.wikimedia.org/r/1125375

QA results

session initiation and search

✅ all good

Screenshot from 2025-02-25 11-15-40.png (1×1 px, 231 KB)

@ngkountas I have a question here, right now, we don't have any entry points enabled so the only way to direct open the special page and init event being logged fine. When we have the other entry points, like article footer, will the action_source be captured accordingly? For the experiment, I think we have article footer, languages selector and search.

This is not happening currently. I recommend creating a separate ticket for this. This task already covers a lot of things.

abi_ changed the task status from In Progress to Open.Mar 17 2025, 6:33 AM

I see the event being logged:

image.png (790×2 px, 260 KB)

abi_ changed the task status from Open to In Progress.Mar 17 2025, 6:33 AM

This is not happening currently. I recommend creating a separate ticket for this. This task already covers a lot of things.

Created: T389062

I see the event human_translation_card being logged on production

image.png (1×1 px, 275 KB)