Page MenuHomePhabricator

[SPIKE] Review measurement plan & instrumentation spec
Closed, ResolvedPublic2 Estimated Story PointsSpike

Description

Background

We'd like to review the measurement plan & instrumentation spec from an engineering perspective and create tickets for instrumentation

Acceptance criteria

Event Timeline

ovasileva triaged this task as High priority.
ovasileva moved this task from Incoming/Inbox to Needs Refinement on the Reader Growth Team board.
HSwan-WMF set the point value for this task to 2.Aug 21 2025, 4:16 PM
HSwan-WMF moved this task from Needs Refinement to Sprint 4 on the Reader Growth Team board.
mfossati changed the task status from Open to In Progress.Aug 27 2025, 2:18 PM
mfossati moved this task from Committed to Doing on the Reader Growth Team (Sprint 4 ) board.

Here's something which @KSarabia-WMF brought to my attention.

The clickTracking code in the WikimediaEvents extension (which our instrumentation may be expected to integrate with) is not really compatible with Vue.js out of the box. The default approach to click tracking uses jQuery to grab all elements which have specifcic data-attributes and attach an event listener to them.

Vue.js (which we are using to build our prototype) is a reactive UI framework, so elements are added and removed from the page constantly based on what the user is doing. Elements added after the inital page load (i.e. our entire ImageBrowsing UI) are not going to receive these handlers by default.

Fortunately, I think there are several ways we can solve this problem. My initial suggestion would be that we should define a custom Vue plugin which will automatically introduce any necessary event listeners for a given component in the mounted lifecycle hook (and should tear them all down in the unmounted hook). By using a plugin, we can ensure that this happens automatically in every component within our UI. Then developers could use normal data-attributes in the templates as they would elsewhere in MediaWiki.

This is just a rough idea of a solution, and we'll need to make sure we understand everything that WikimediaEvents is doing to ensure we can implement similar functionality here. Ideally we could get someone from the Experiment Platform team to weigh in. Maybe this should be a spike task for next sprint.

Change #1182672 had a related patch set uploaded (by Kimberly Sarabia; author: Kimberly Sarabia):

[mediawiki/extensions/ReaderExperiments@master] [wip] sample of event tracking in carousel

https://gerrit.wikimedia.org/r/1182672

@mfossati @KSarabia-WMF could you summarize the meeting you had this morning here for Olga's benefit? Thanks in advance!

The main action item is a request for confirmations. Pasting the message I sent on Slack for reference:
Hey @ovasileva, yesterday we met with @jwang to review image browsing’s instrumentation plan & spec. I’d like to double-check with you the following points:

  1. primary metric - click-through rate
    • it shall be computed *within the image browser* - # users who clicked on an image in the carousel / # users exposed to the image carousel
    • it’s not subject to A/B testing - the control group isn’t needed
  2. engagement interactions don’t need A/B testing neither - we just want to analyze the interactions within the feature
  3. A/B testing holds for guardrail metrics - retention rate (to be compared with the A/A baseline) & session length
  4. I couldn’t find the list of target Wikipedias in the instrumentation plan. Can you please re-confirm?

Additional comments:

  • we should also track clicks on wikilinks that show up in captions
  • we might want to group experiment data by article type (e.g., person, place, etc.) for fine-grained understanding of the feature's performance
  • horizontal or vertical scrolling (carousel or VTOC respectively) could be tracked, although session length can be thought as a signal that encompasses scrolling
Jdlrobson-WMF added a subscriber: mfossati.

This ticket can likely be closed. @egardner will take care of that.

Restricted Application changed the subtype of this task from "Task" to "Spike". · View Herald TranscriptSep 12 2025, 8:09 PM

Change #1182672 abandoned by Kimberly Sarabia:

[mediawiki/extensions/ReaderExperiments@master] [wip] sample of event tracking in carousel

https://gerrit.wikimedia.org/r/1182672