@Iflorez @nettrom_WMF, I left some comments in the instrumentation spec. Here are some questions that can be relevant for the implementation:

• GrowthExperiments uses analytics/legacy/newcomertask schema to log impressions of the offered tasks in the homepage (one per card impression) and in the post edit dialog. Is this schema suitable to log impressions of the add-link suggestions in read mode?
  • [If yes to the prior question] The newcoemrtask schema contains a property newcomer_task_token that allows to relate the impression events with the events from the specific instrumentation of the structured task (analytics/mediawiki/structured_task/article/link_suggestion_interaction schema), akin to homepage_pageview_token which allows to relate homepage visits with structured tasks editing sessions.
• The instrumentation spec states that the Number of impressions of an article with the task needs to be tracked two times? One through non-MP instrument (row C7) and one through MP instrument (row C8). Also, the click event in row C9, should be captured with MP instruments, is that correct?
  • [If yes to the prior question] Do we need a "relation token" between the MP instruments and the non-MP ones?
• Engineers have raised concerns about the risk of collecting user ids/names and page ids/titles as a privacy leak / security threat that could be exploited to know "what a user reads" (originally brought by @Urbanecm). As instrument/schemas are set now that user data would be indeed collected. I believe it's never come up as GrowthExperiments has few features in read mode (maybe Help panel?). Is this a concern for the L3SC review or in general?
  • [If yes to the prior question] Engineers have proposed ways to avoid collecting sensitive user data by dropping the page title/id or user data (id, name, client ip, etc.) from the event payload. Does this sound acceptable from the data collection pov to do further analysis?

GrowthExperiments uses analytics/legacy/newcomertask schema to log impressions of the offered tasks in the homepage (one per card impression) and in the post edit dialog. Is this schema suitable to log impressions of the add-link suggestions in read mode?

I see that it has the following fields which are important:

• newcomer_task_token
• task_type
• page_id
• page_title

If we're only showing 1 task per username, in keeping with the set quality gates, then the above works.
Also, if we're not looking for additional fidelity on impressions then the above works.

The instrumentation spec states that the Number of impressions of an article with the task needs to be tracked two times? One through non-MP instrument (row C7) and one through MP instrument (row C8). Also, the click event in row C9, should be captured with MP instruments, is that correct?

Impressions can be tracked once (C7). They are listed again in C8 as they are a component of calculating CTR. I've updated the C8 mention to note "[from above; not tracked twice]". I'm happy to follow an improved method that is less confusing.

Also, the click event in row C9, should be captured with MP instruments, is that correct?

I understand from the 10/31/24 meeting that we will not be using the Metrics Platform. I've updated the entire sheet so none of the rows show MP designation.

Engineers have proposed ways to avoid collecting sensitive user data by dropping the page title/id or user data (id, name, client ip, etc.) from the event payload. Does this sound acceptable from the data collection pov to do further analysis?

Thank you for this; I would be happy to learn more and to avoid collecting sensitive user data.
I'm also wondering if the above schema noted in point 1 will suffice.

Change #1092848 had a related patch set uploaded (by Sergio Gimeno; author: Sergio Gimeno):

[operations/mediawiki-config@master] [DNM] PoC: newcomer task stream config

https://gerrit.wikimedia.org/r/1092848

Change #1092853 had a related patch set uploaded (by Sergio Gimeno; author: Sergio Gimeno):

[mediawiki/extensions/GrowthExperiments@master] [DNM][WIP] PoC: instrument surfaced link recommendation in read mode

https://gerrit.wikimedia.org/r/1092853

Change #1092897 had a related patch set uploaded (by Michael Große; author: Michael Große):

[mediawiki/extensions/GrowthExperiments@master] surfacing: prepare tracking visible highlights

https://gerrit.wikimedia.org/r/1092897

Change #1092897 merged by jenkins-bot:

[mediawiki/extensions/GrowthExperiments@master] surfacing: prepare tracking visible highlights

https://gerrit.wikimedia.org/r/1092897

We just talked about this in Estimation/SprintPlanning and there still does not seem to be clarity about how to move forward with this and which schema to use.

From what I gathered from the conversation, there seem to be (at least) 3 options:

1. Modify analytics/legacy/newcomertask
   • as I understand it, this schema is designed for tracking the interaction with the newcomer tasks on Special:Homepage, and we would need to somehow extend it to also apply to reading-mode suggestions
2. Modify analytics/mediawiki/structured_task/article/link_suggestion_interaction/
   • as I understand it, this schema is designed for tracking the interaction with a link-suggestion inside a VisualEditor reading mode session, and we would need to somehow extend it to also apply to reading-mode suggestions
3. Create a new schema, maybe by copying one of the other two, that is specific to reading-mode suggestions and that we can adjust without polluting the existing specialized schemas.

I think, I would favor creating a new schema for this and moving from there, because we're least likely to break anything in a hard-to-fix way. But I'm not sure how much overhead that would be on the MetricsPlatform/data-analytics side of things.
To me, it seems like just:

• copy&paste a schema,
• adjust the name and version number
• remove the active_interface and events that we don't need, add the ones we do
• maybe ask someone to deploy it somewhere
• Done ✅

Thoughts?

In T377097#10359007, @Michael wrote:

1. Modify analytics/mediawiki/structured_task/article/link_suggestion_interaction/
   • as I understand it, this schema is designed for tracking the interaction with a link-suggestion inside a VisualEditor reading mode session, and we would need to somehow extend it to also apply to reading-mode suggestions

Per conversation with engineers, the preferred approach is (2), mainly because it already contains most of the possible interactions with link suggestions in an article page in reading mode (very similar to the interactions with VE in the structured task session). The schema will be updated to contain two new active interfaces and necessary new actions (some can be reused). For instance:

Also, for deeper feature performance analysis, we'd like to track some extra metrics using intruments that are totally user agnostic, just a number:

• A user clicked “No” on the surfaced suggestions popup
• A user clicked the “X”-Button on the surfaced suggestions popup
• A user closed the popup by clicking outside
• A user closed the popup by scrolling away
• A surfaced suggestion on a page was interacted with

Does this sound accurate? If so, I can file a separate task for it. cc @Michael

Change #1098607 had a related patch set uploaded (by Michael Große; author: Michael Große):

[mediawiki/extensions/GrowthExperiments@master] feat(surfacing): track performance metrics

https://gerrit.wikimedia.org/r/1098607

Sounds good! Here is the change for the performance metrics: feat(surfacing): track performance metrics

sgimeno opened https://gitlab.wikimedia.org/repos/data-engineering/schemas-event-secondary/-/merge_requests/31

link_suggestion_interaction: add read mode interfaces and actions

Change #1098906 had a related patch set uploaded (by Sergio Gimeno; author: Sergio Gimeno):

[mediawiki/extensions/GrowthExperiments@master] instrumentation(StructuredTaskSurfacer): add read mode interaction tracking

https://gerrit.wikimedia.org/r/1098906

Change #1098607 merged by jenkins-bot:

[mediawiki/extensions/GrowthExperiments@master] feat(surfacing): track performance metrics

https://gerrit.wikimedia.org/r/1098607

Change #1098906 merged by jenkins-bot:

[mediawiki/extensions/GrowthExperiments@master] instrumentation(StructuredTaskSurfacer): add read mode interaction tracking

https://gerrit.wikimedia.org/r/1098906

Change #1099681 had a related patch set uploaded (by Michael Große; author: Michael Große):

[mediawiki/extensions/GrowthExperiments@wmf/1.44.0-wmf.5] surfacing: prepare tracking visible highlights

https://gerrit.wikimedia.org/r/1099681

Change #1099684 had a related patch set uploaded (by Michael Große; author: Michael Große):

[mediawiki/extensions/GrowthExperiments@wmf/1.44.0-wmf.5] feat(surfacing): track performance metrics

https://gerrit.wikimedia.org/r/1099684

Change #1099685 had a related patch set uploaded (by Michael Große; author: Sergio Gimeno):

[mediawiki/extensions/GrowthExperiments@wmf/1.44.0-wmf.5] instrumentation(StructuredTaskSurfacer): add read mode interaction tracking

https://gerrit.wikimedia.org/r/1099685

urbanecm merged https://gitlab.wikimedia.org/repos/data-engineering/schemas-event-secondary/-/merge_requests/31

link_suggestion_interaction: add read mode interfaces and actions

Change #1099681 abandoned by Michael Große:

[mediawiki/extensions/GrowthExperiments@wmf/1.44.0-wmf.5] surfacing: prepare tracking visible highlights

Reason:

Squashed into Ic1fa23a1eeaba34f74644ad222e50559d605a74c

https://gerrit.wikimedia.org/r/1099681

Change #1099684 abandoned by Michael Große:

[mediawiki/extensions/GrowthExperiments@wmf/1.44.0-wmf.5] feat(surfacing): track performance metrics

Reason:

Squashed into Ic1fa23a1eeaba34f74644ad222e50559d605a74c

https://gerrit.wikimedia.org/r/1099684

Change #1099685 abandoned by Michael Große:

[mediawiki/extensions/GrowthExperiments@wmf/1.44.0-wmf.5] instrumentation(StructuredTaskSurfacer): add read mode interaction tracking

Reason:

Squashed into Ic1fa23a1eeaba34f74644ad222e50559d605a74c

https://gerrit.wikimedia.org/r/1099685

In T377097#10362920, @Sgs wrote:

Per conversation with engineers, the preferred approach is (2), mainly because it already contains most of the possible interactions with link suggestions in an article page in reading mode (very similar to the interactions with VE in the structured task session). The schema will be updated to contain two new active interfaces and necessary new actions (some can be reused). For instance:

action	active_interface	Description
pseudo-impression	readmode_page	Impression of the page with recommendations
impression	readmode_page	Impression of the first highlighted text when seen by the user (inside viewport)
suggestion_accept	readmode_suggestion_dialog	Click on the "Yes" button of one suggestion popup

....
Does this sound accurate? If so, I can file a separate task for it. cc @Michael

@Sgs

1. Can you share the follow up task?
2. I want to make sure that I'm looking for the right data. For 1.2.6, should I be looking for data where active_interface is in

('readmode_suggestion_dialog', 'readmode_page')

and for one of these below-listed actions (to capture impressions and yes clicks)?

viewport_impression # A link suggestion highlight was displayed in the user's viewport in the readmode_article_page
suggestion_accept_yes # The user pressed "Yes" in the readmode_suggestion_dialog

And how is a regular view action labeled? impression?

3. Are we limiting the number of users that a task is shown to?

I re-checked the workflows for Surfacing add link for eswiki, frwiki, and arzwiki

• new accounts see the highlighted link suggestions
• In the console (with mw.loader.using( 'mediawiki.api' ).then( function () { new mw.Api().saveOption( 'eventlogging-display-web', '1' ); } );) only three following events are displayed:

{action: 'impression', action_data: '', homepage_pageview_token: '60d8cf98-b97a-4f73-8d66-2f84b56d0045', newcomer_task_token: '39a2c3f6-82a6-4007-a6df-6f5eef86da14', is_mobile: true, …}
$schema: "/analytics/mediawiki/structured_task/article/link_suggestion_interaction/1.8.0"
action: "impression"
action_data: ""
active_interface: "readmode_article_page"
dt: "2024-12-06T21:47:41.248Z"
homepage_pageview_token: "60d8cf98-b97a-4f73-8d66-2f84b56d0045"
is_mobile: true
meta: 
{stream: 'mediawiki.structured_task.article.link_suggestion_interaction', domain: 'test.wikipedia.org'}
newcomer_task_token: "39a2c3f6-82a6-4007-a6df-6f5eef86da14"
page_id: 114589
page_title: "Earthquake prediction"

{action: 'viewport_impression', action_data: '', homepage_pageview_token: '60d8cf98-b97a-4f73-8d66-2f84b56d0045', newcomer_task_token: '39a2c3f6-82a6-4007-a6df-6f5eef86da14', is_mobile: true, …}
$schema: "/analytics/mediawiki/structured_task/article/link_suggestion_interaction/1.8.0"
action: "viewport_impression"
action_data: ""
active_interface: "readmode_article_page"
dt: "2024-12-06T21:51:56.560Z"
homepage_pageview_token: "60d8cf98-b97a-4f73-8d66-2f84b56d0045"
is_mobile: true
meta: 
{stream: 'mediawiki.structured_task.article.link_suggestion_interaction', domain: 'test.wikipedia.org'}
newcomer_task_token: "39a2c3f6-82a6-4007-a6df-6f5eef86da14"
page_id: 114589
page_title: 
"Earthquake prediction"

{action: 'suggestion_accept_yes', action_data: '', homepage_pageview_token: 'bada6c39-c003-41ae-859e-adaed0ce1da7', newcomer_task_token: '3e2ed1ad-a469-4efa-b5f3-d8bc4e028d03', is_mobile: true, …}

• but on the grafana board grafana for testwiki clicks on highlight are present and the count, as far as I could see, is accurate.

In T377097#10384979, @Iflorez wrote:

@Sgs

1. Can you share the follow up task?

I ended labeling the related changes with this task, no new task was created, here are the relevant changes: link_suggestion_interaction/1.8.0.yaml

2. I want to make sure that I'm looking for the right data. For 1.2.6, should I be looking for data where active_interface is in

('readmode_suggestion_dialog', 'readmode_page')

and for one of these below-listed actions (to capture impressions and yes clicks)?

viewport_impression # A link suggestion highlight was displayed in the user's viewport in the readmode_article_page
suggestion_accept_yes # The user pressed "Yes" in the readmode_suggestion_dialog

And how is a regular view action labeled? impression?

Apologies, the final names in the changes we've made are a little different from the proposed table in due to technical constraints, these are the relevant actions/interfaces with the final naming:

Does this help?

3. Are we limiting the number of users that a task is shown to?

We limit the feature to show only for registered users, on mobile page views, and with 0 edits. But we do not limit the number of users from that cohort in a sampling sense. Sadly the numbers are pretty low.

In T377097#10388024, @Etonkovidova wrote:

• but on the grafana board grafana for testwiki clicks on highlight are present and the count, as far as I could see, is accurate.

That is expected, the grafana dashboard has some extra metrics that we're not sending the the analytics data lake. If we deem them relevant we can add them, but they were not considered necessary for the initial analysis of activation rates.

@Sgs Thank you for clarifying and confirming final names

Apologies, the final names in the changes we've made are a little different from the proposed table in due to technical constraints, these are the relevant actions/interfaces with the final naming:

action	active_interface	Description
impression	readmode_article_page	Impression of a page with recommendations, "regular view"
viewport_impression	readmode_article_page	Impression of the first highlighted text when seen by the user (inside viewport)
suggestion_accept_yes	readmode_suggestion_dialog	Click on the "Yes" button of one suggestion popup. (As opposed of suggestion_accept,recommendedlinktoolbar_dialog which is the "real" accept of the suggestion in editing mode)

And I understand that the below are instrumented and are not on the schema so they are not found in the data lake but they are appearing on the Grafana board
A user clicked “No” on the surfaced suggestions popup - instrumented and on Grafana
A user clicked the “X”-Button on the surfaced suggestions popup - instrumented and on Grafana
A user closed the popup by clicking outside - instrumented and on Grafana
Clicks on highlight - instrumented and on Grafana
A surfaced suggestion on a page was interacted with - ?

In T377097#10391023, @Iflorez wrote:

[...]
A surfaced suggestion on a page was interacted with - ?

In retrospect, I'm not actually sure what that means. We have performance metrics and (some) analytics instrumentation for clicking on the highlight, and then clicking Yes/No/X/outside of/on the popup. Are we missing anything here?

I double-checked eventlogging in wmf.6 to verify the following

• the 'impression' event is recorded even if a highlighted suggestion is in the collapsed article section - true
• the viewport_impression appears only when a section is un-collapsed

I've marked QA Engineer tests as Done. For other testing or follow-ups the task is moved to Test in Production column.

@KStoller-WMF Do we need to do anything else to finish the other two A/Cs that are unchecked?

@KStoller-WMF Thank you for the follow-up and my apologies for my delay.
No remaining open questions from my end.

Change #1092853 abandoned by Sergio Gimeno:

[mediawiki/extensions/GrowthExperiments@master] [DNM][WIP] PoC: instrument surfaced link recommendation in read mode

Reason:

Done with different appraoch in Ia129f35255893dc2e14add338835978076d75865

https://gerrit.wikimedia.org/r/1092853

Change #1092848 abandoned by Sergio Gimeno:

[operations/mediawiki-config@master] [DNM] PoC: newcomer task stream config

Reason:

Done in Ia129f35255893dc2e14add338835978076d75865 without needing to create a new stream

https://gerrit.wikimedia.org/r/1092848