Page MenuHomePhabricator

[SPIKE] Define/refine experiment-related instrumentation API
Closed, ResolvedPublic8 Estimated Story PointsSpike

Description

I've been thinking about the feedback that we've received on the draft API proposal thus far. AIUI the main sticking point has been about ClickThroughRateInstrument. The feedback about it has been:

  1. It's atypical: Instrument developers either embed instruments in their feature code or they embed hooks in their feature code which instruments embedded in other modules can listen to. ClickThroughRateInstrument takes a different take altogether, listening to DOM events directly. Kosta called this "outside in" instrumenting (paraphrasing) whereas instrument developers typically take an "inside out" approach
  2. It's (currently) incompatible with Vue apps: ClickThroughRateInstrument requires a stable DOM element to function. While Vue does an excellent job at maintaining DOM elements during the mount/patch render phase, there's no guarantee that a DOM element is stable during the lifetime of the Vue app

So how do we reconcile The Vision™ – pre-canned, specified, well-documented instruments that can be used as Product Health Metrics as well as primary, secondary, and guardrail metrics – with the status quo? We offer the components of the instrument as well as the instrument itself.

An instrument has three responsibilities:

  1. Manage state
  2. Create analytics events
  3. Respond to UX events

We can offer a low-level component for each of the above that instrument developers can use as well as a high-, and mid-level components that uses those components.

Example 1

Consider a feature that shows a dialog with two buttons labelled "Accept" and "Reject." The experiment owner wants to run an experiment on the dialog and wants to monitor the click-through rate of the buttons.

// NOTE: Object and method names are all subject to intense bikeshedding

const ACCEPT = {
	friendly_name: E1_FRIENDLY_NAME,
	selector: E1_SELECTOR
};

const DECLINE = {
	friendly_name: E2_FRIENDLY_NAME,
	selector: E2_SELECTOR
};

const eventStream = mw.xLab.newEventStream( STREAM_ID, SCHEMA_ID );



// High Level
// ==========
const {
	Instrument
} = require( 'ext.wikimediaEvents.xLab' ).ClickThroughRate;

Instrument.start( eventStream, ACCEPT, DECLINE );

// Or...
eventStream.include( Instrument, ACCEPT, DECLINE );



// Mid Level
// =========
const {
	EventSubmitter
} = require( 'ext.wikimediaEvents.xLab' ).ClickThroughRate;

const eventSubmitter = new EventSubmitter( eventStream );

// When the feature code shows the dialog:
eventSubmitter.submitImpression( ACCEPT, DECLINE /*, ... */ );

// When the feature code detects that the user has clicked the Accept button:
eventSubmitter.submitClick( ACCEPT );



// Low Level
// =========
const {
	StateManager,
	EventFactory
} = require( 'ext.wikimediaEvents.xLab' ).ClickThroughRate;

let state = StateManager.newState();

// When the feature code shows the dialog:
state = StateManager.impression( state, ACCEPT );

eventStream.submitInteraction( ...EventFactory.impression( state, ACCEPT ) );

state = StateManager.impression( state, DECLINE );

eventStream.submitInteraction( ...EventFactory.impression( state, DECLINE ) );

// When the feature code detects that the user has clicked the Accept button:
state = StateManager.click( state, ACCEPT );

eventStream.submitInteraction( ...EventFactory.click( state, ACCEPT ) );

Event Timeline

Restricted Application changed the subtype of this task from "Task" to "Spike". · View Herald TranscriptApr 2 2025, 9:30 AM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript

Change #1135712 had a related patch set uploaded (by Phuedx; author: Phuedx):

[mediawiki/extensions/MetricsPlatform@master] Experiment: Add #isAssignedGroup()

https://gerrit.wikimedia.org/r/1135712

Change #1135712 merged by jenkins-bot:

[mediawiki/extensions/MetricsPlatform@master] Experiment: Add #isAssignedGroup()

https://gerrit.wikimedia.org/r/1135712