Page MenuHomePhabricator

[SPIKE] Prototype API for Submitting Core Interaction Events
Closed, ResolvedPublic8 Estimated Story PointsSpike

Description

Background/Goal

We have audited the schemas of the top 15 most active analytics instruments and have prototyped so-called Core Interaction schema fragments and concrete schemas. We now need to prototype an API for submitting Core Interaction events.

KR/Hypothesis (Initiative)

If we develop a centralized experimentation platform that can define, deploy, and get feedback on experiments, Product teams will develop and instrument their features for experimentation.

See also SDS2.5.1 Centralized experimentation platform in Asana.

Acceptance Criteria

  • We have agreed on an API for tracking the performer clicking a UI element
  • We have agreed on an API for entering the performer into a funnel
  • The instrumentation developer can submit an event with or without top-level custom data
  • The instrumentation developer can still use the existing APIs

Required

Notes

  1. When the performer has entered into a funnel, then the library should record the position of the event in the sequence of events sent in that funnel, e.g.
{ action_type: "init", funnel_event_sequence_position: 1 }
{ action_type: "click", action_subtype: "open", action_source: "sidebar", funnel_event_sequence_position: 2 }
{ action_type: "click", action_subtype: "close", action_source: "sidebar", funnel_event_sequence_position: 3 }
  1. Every method MUST be able to accept a schema ID

Questions

  1. How long can a funnel last? Does this vary by platform?

Event Timeline

phuedx renamed this task from [EPIC] Prototype Metrics Platform API for submitting Core Interaction events to [EPIC] Prototype Metrics Platform API for Submitting Core Interaction Events.Sep 1 2023, 3:16 PM
phuedx triaged this task as High priority.
phuedx added projects: Epic, Test Kitchen.
phuedx updated the task description. (Show Details)
phuedx added a subscriber: VirginiaPoundstone.
phuedx renamed this task from [EPIC] Prototype Metrics Platform API for Submitting Core Interaction Events to [SPIKE] Prototype Metrics Platform API for Submitting Core Interaction Events.Sep 1 2023, 3:41 PM
phuedx edited projects, added Spike; removed Epic.
phuedx updated the task description. (Show Details)
Restricted Application changed the subtype of this task from "Task" to "Spike". · View Herald TranscriptSep 1 2023, 3:41 PM

A very rough proposal from the parent epic:

type Token = string;

interface ScopedEventSubmitter {
	readonly funnelEntryToken: Token;
	readonly funnelEventSequenceID: integer;

	submitClick( elementID: string, elementFriendlyName?: string ): void;
}

interface MetricsPlatform {

	/**
	 * Creates an event submitter that submits events to the stream. All
	 * submitted events will have the `funnel_entry_token` property set.
	 *
	 * Before the event submitter is returned, a `funnel_enter` event
	 * representing the user entering the funnel is submitted.
	 */
	enterFunnel( streamName: string, funnelEntryToken?: Token ): ScopedEventSubmitter;

	getEnteredFunnels(): string[];
}

namespace mw {
	export metricsPlatform: MetricsPlatform;
}

// ---

const PAGEVIEW_TOKEN: Token = mw.user.getPageviewToken();

const submitter = mw.metricsPlatform.enterFunnel( 'mediawiki.ui.clicks', PAGEVIEW_TOKEN );

// Later...
submitter.submitClick( 'pt-notifications-alert', 'notifications' );
phuedx renamed this task from [SPIKE] Prototype Metrics Platform API for Submitting Core Interaction Events to [SPIKE] Prototype API for Submitting Core Interaction Events.Sep 12 2023, 1:33 PM
phuedx updated the task description. (Show Details)

Based on the initial core interactions schemas, and in addition to the initial proposal above, can the following work for click, view, and funnel events?

submitMetricsEvent(String schemaId, String stream, <Data Object> interactionData, ?Map<String, Object> customData) {
	...
	// Add contextual attribute data for every Metrics Event.
}

submitClick(String stream, <Data Object> interactionData, ?Map<String, Object> customData) {
	...
	event.action = "click";
        event.meta.schemaId = "/analytics/metrics_platform/{app,web}/click/1.0.0";
}

submitView(String stream, <Data Object> interactionData, ?Map<String, Object> customData) {
	...
	event.action = "view";
        event.meta.schemaId = "/analytics/metrics_platform/{app,web}/view/1.0.0";
}

submitInteraction(String schemaId, String stream, <Data Object> interactionData, ?Map<String, Object> customData) {
	...
	event.action = interactionData.action;
        event.meta.schemaId = schemaId;
}

Per ScopedEventSubmitter interface in T345439#9137016

enterFunnel(String stream, Token funnelEntryToken, <Data Object> interactionData, ?Map<String, Object> customData): ScopedEventSubmitter {
	...
	event.action = "funnel_enter";
        event.meta.schemaId = "/analytics/metrics_platform/{app,web}/{click,view}/1.0.0";
}

Do we need a way to mark the exit of a funnel?

exitFunnel(String stream, Token funnelEntryToken, <Data Object> interactionData, ?Map<String, Object> customData): ScopedEventSubmitter {
	...
	event.action = "funnel_exit";
        event.meta.schemaId = "/analytics/metrics_platform/{app,web}/{click,view}/1.0.0";
}

For PHP + Javascript clients, a Data Object type of parameter would presumably be needed to capture the common interaction properties for a given event. The Metrics Client object is instantiated with the contextual attribute data (web fragment) that is available for every event. For these clients, the parameters for the different type of submit methods include interaction data (to populate the common interaction properties only) with optional custom data alongside schema id and stream name.

In the case of the Java + Swift clients, some of the contextual attributes (i.e. page and performer data objects) are dynamic and need to be submitted with the event if/when they become available. For these clients, the interactionData object would be bundled with the core contextual attribute data (app fragment) and passed into the submit methods alongside optional custom data, schema id, and stream name.

For all clients, custom data would need to be parsed and serialized with each event as top level properties in order to validate against the specified schema id and stream name passed in with the corresponding event. The value of each custom property would be limited to primitive types: string, integer, boolean, null. Each client library will have to validate the allowable types of the custom data parameter if/when it exists. Owners of custom interaction schemas will be responsible for passing the correct data types in the custom data map parameter from client code so that events validate successfully.

Presumably the API will grow and evolve over time as more core interactions are proposed and developed.

@phuedx does this suffice for an initial short list of submit methods?

submitMetricsEvent(String schemaId, String stream, <Data Object> interactionData, ?Map<String, Object> customData) {
	...
	// Add contextual attribute data for every Metrics Event.
}

I'd be careful about re-using the term "metrics event" here because the monoschema's title is /analytics/mediawiki/client/metrics_event. With this in mind, is this a duplicate of submitInteraction() below?

submitClick(String stream, <Data Object> interactionData, ?Map<String, Object> customData) {
...
event.action = "click";

event.meta.schemaId = "/analytics/metrics_platform/{app,web}/click/1.0.0";

}
<snip />

All of these LGTM. Remember that we set event.$schema and not event.meta.schemaId though 🙂

Do we need a way to mark the exit of a funnel?

exitFunnel(String stream, Token funnelEntryToken, <Data Object> interactionData, ?Map<String, Object> customData): ScopedEventSubmitter {
	...
	event.action = "funnel_exit";
        event.meta.schemaId = "/analytics/metrics_platform/{app,web}/{click,view}/1.0.0";
}

Good catch. Yes.

The value of each custom property would be limited to primitive types: string, integer, boolean, null. Each client library will have to validate the allowable types of the custom data parameter if/when it exists. Owners of custom interaction schemas will be responsible for passing the correct data types in the custom data map parameter from client code so that events validate successfully.

I think it's enough to say that custom data (instrument-specific data?) MUST be serializable and MUST validate against a schema that follows the rules.

@phuedx does this suffice for an initial short list of submit methods?

It's a good start 👍 I've marked all of the new methods in the JS client library as unstable as they'll be subject to change while we test their integration with a handful of existing instruments.

Looks good to me in general!
Could we have interactionData and customData be the same parameter maybe?
Does it matter for them to be separate?

thanks for the feedback

I'd be careful about re-using the term "metrics event" here because the monoschema's title is /analytics/mediawiki/client/metrics_event. With this in mind, is this a duplicate of submitInteraction() below?

I have a submitMetricsEvent() method in the Java client for adding/formatting all the parameterized and meta data for the event. It's the analog to the PHP + Javascript clients' dispatch() method.

I was thinking that submitInteraction() could be the base level event submission for the common interaction schema in case there's a need to use that schema for event types not yet supported by MP?

Remember that we set event.$schema

touché

it's enough to say that custom data (instrument-specific data?) MUST be serializable and MUST validate against a schema that follows the rules

great


Another attempt at an initial list of MP methods:

// Whether it's called "dispatch()" or "submitMetricsEvent()", this method formats the event for submission to the batch queue for each client.
submitMetricsEvent(
   String schemaId,
   String stream,
   String eventName,
   <Data Object> clientData, // for app clients - web clients have this data available at instantiation of the metrics client object.
   ?<Data Object> interactionData,
   ?Map<String, Object> customData
) {
	...
	event.action = eventName;
        event.$schema = schemaId;
}

submitClick(String stream, <Data Object> interactionData, ?Map<String, Object> customData) {
	...
	event.action = "click";
        event.$schema = "/analytics/metrics_platform/{app,web}/click/1.0.0";
}

submitView(String stream, <Data Object> interactionData, ?Map<String, Object> customData) {
	...
	event.action = "view";
        event.$schema  = "/analytics/metrics_platform/{app,web}/view/1.0.0";
}

submitInteraction(String schemaId, String stream, <Data Object> interactionData, <Data Object> clientData, ?Map<String, Object> customData) {
	...
	event.action = interactionData.action;
        event.$schema = schemaId;
}

Adding more from the event types proposal spreadsheet (see T345729):

submitHover() {
	...
	event.action = "hover";
        event.$schema = "/analytics/metrics_platform/{app,web}/{hover}/1.0.0";
}

submitScroll() {
	...
	event.action = "scroll";
        event.$schema = "/analytics/metrics_platform/{app,web}/{scroll}/1.0.0";
}

submitStateSummary() {
	...
	event.action = "state_summary";
        event.$schema = "/analytics/metrics_platform/{app,web}/{state_summary}/1.0.0";
}

submitInstall() {
	...
	event.action = "install";
        event.$schema = "/analytics/metrics_platform/{app,web}/{install}/1.0.0";
}

Is a separate init event needed?

submitInit() {
	...
	event.action = "init";
        event.$schema = "/analytics/metrics_platform/{app,web}/{init}/1.0.0";
}

Per ScopedEventSubmitter interface in T345439#9137016

enterFunnel(
   String stream, 
   Token funnelEntryToken, 
   <Data Object> interactionData, 
   ?Map<String, Object> customData
): ScopedEventSubmitter {
	...
	event.action = "funnel_enter";
        event.$schema = "/analytics/metrics_platform/{app,web}/{click,view}/1.0.0";
}

exitFunnel(
   String stream, 
   Token funnelEntryToken, 
   <Data Object> interactionData, 
   ?Map<String, Object> customData
): ScopedEventSubmitter {
	...
	event.action = "funnel_exit";
        event.$schema = "/analytics/metrics_platform/{app,web}/{click,view}/1.0.0";
}

Could we have interactionData and customData be the same parameter maybe?
Does it matter for them to be separate?

Since customData is optional, i was thinking to keep it a separate data object parameter from interactionData (which has well-defined properties). And because it can have variable values (limited to allowed enums), each client library has to parse the customData map/collection passed into a given submit method and serialize the associated event to transform the custom data to become top-level properties for validation against its corresponding schema.

Java library has an added data object parameter for the client (common/core contextual attributes) data since they are variable/dynamic and not always available at the time of metrics client instantiation.

Curious if it's better practice to consolidate the data object parameters?
Feels intuitively like it could be confusing but my tendency is to over-complicate things so ¯\_(ツ)_/¯

Curious if it's better practice to consolidate the data object parameters?

I'd argue that it is.

Generally speaking, the design decisions of a module (package, class, method) should be hidden from external collaborators. This is called information hiding. The greater the number of parameters, the greater the chance that your design decisions are visible.

One way of reducing the number of parameters is introducing a parameter object. In this case, you might consider introducing a InteractionEvent base class:

submitInteraction(
  String stream,
  String schemaID,
  InteractionEvent interactionData
) {
  /* ... */
}

@Data
class InteractionData {
  @Getter private string actionName;
  @Getter private string actionSubType;
  @Getter @Setter private string actionContext;
  @Getter @Setter private string actionSource;

  // The Kitchen Sink™
  //
  // TODO: Rename this to instrumentData?
  @Getter @Setter private Map<String, Object> customData;

  public InteractionData( String action  ) {
    this.action = action;
  }

  public InteractionData( String action, String actionSubType ) {
    this.action = action;
    this.actionSubType = actionSubType;
  }

  public static InteractionData click() {
    return InteractionData( 'click' /* , ... */ );
  }
}

// The accompanying serializer, which handles serializing the top-level data that MP knows
// about and the top-level data that the instrument knows about.
class InteractionDataSerializer implements JsonSerializer<InteractionData> {
  @Override
  public JsonElement serialize(Student src, Type typeOfSrc, JsonSerializationContext context) {
    /* ... */
  }
}

I think I'm going to follow my own advice here and update the MR for T346287: [Javascript] Create Metrics Platform API for Submitting Core Interaction Events.

I've been bold and moved this to Sign Off as I think we're happy with the overall shape of the API, including the first three parameters to submitInteraction(), but are tweaking the names of the interaction and/or custom data objects, which can just as easily be worked through during code review.

When we've finalised the APIs, I'll update this task and move it to Done.