Page MenuHomePhabricator

[User Story] Partial migration of *UIActions instrument to the Core Interaction API
Closed, ResolvedPublic8 Estimated Story Points

Description

Description

The *UIActions instrument is a good case study for us (Data Products) as it logs custom data (i.e. instrument- or domain-specific data) that would have to be schemaed and we're going to need a documented process for this as we onboard other teams. However, the Web team, who own the instrument, are currently updating it in order to establish baseline metrics prior to releasing a new feature.

In order to minimise risk for the Web team, let's do a parallel implementation of the logging part of the instrument that uses the Core Interaction API.

User Story/ies

As a developer, I want/need to understand the process of migrating my instrument to the Metrics Platform, so I can migrate my remaining instruments without intervention.

Outcome

  • The new instrument is publishing events to distinct stream in parallel with the original

Required

  • Unit tests
  • Integration tests
  • Testing Instructions
  • Passed QA
  • Documentation

Technical Notes

As noted in the description, the instrument logs custom data. These data are:

  • skinVersion@Jdlrobson: Is this needed?
  • isSidebarCollapsed
  • viewportSizeBucket
  • font
  • is_full_width
  • is_page_preview_on
  • is_pinned
  • is_media_viewer_enabled
  • is_dark_mode_prepared_by_os
  • is_dark_mode_on

Fortunately, these data are already schemaed by the fragment/analytics/web_accessibility/1.0.0 schema fragment. Thus, our first step will be to create a concrete schema for the new event stream that extends (?) analytics/product_metrics/web/base/1.0.0 and mixes in that fragment.

Next, we would need to update WikimediaEvents/modules/ext.wikimediaEvents/desktopWebUIActions.js and WikimediaEvents/modules/ext.wikimediaEvents/mobileWebUIActions.js as follows:

mw.eventLog.logEvent( 'DesktopWebUIActionsTracking', webA11ySettingsData );

// =>

delete webA11ySettingsData.action;

// … and delete all data that can be mixed in by the JS Metrics Platform client library.

mw.eventLog.submitInteraction(
  'mediawiki.web_ui_actions.migrated',
  '/analytics/product_metrics/web/ui_actions/1.0.0',
  action,
  webA11ySettingsData
);

Finally, we'd need to define the corresponding stream in operations/mediawiki-config/wmf-config/ext-EventStreamConfig.php as follows:

ext-EventStreamConfig.php
'mediawiki.web_ui_scroll' => [
  'schema_title' => 'analytics/product_metrics/web/web_base',
  'destination_event_service' => 'eventgate-analytics-external',
  'producers' => [
    'metrics_platform_client' => [

      // See also https://github.com/wikimedia/operations-mediawiki-config/blob/6e1ea777ed3b4e4ec151a4ea9d318cfc0c999784/wmf-config/ext-EventStreamConfig.php#L980
      'provide_values' => [
        'page_namespace',
        'performer_is_logged_in',
        'performer_session_id',
        'performer_pageview_id',
        'performer_edit_count_bucket',
        'mediawiki_skin',
        'mediawiki_database'
      ],
      'curation' => [
        'mediawiki_skin' => [
          'in' => [ 'minerva', 'vector', 'vector-2022' ],
        ],
      ],
  ],
  'sample' => [
    'unit' => 'pageview',
    'rate' => ,
  ],
],

Dependencies

N/A

Artifacts & Resources

N/A

Event Timeline

phuedx created this object with edit policy "All Users".

@phuedx
cc: @VirginiaPoundstone

  • Just wanted to let you know we've finished click tracking instrumentation for the new ClientPreferences feature.
  • As discussed, the clientPref instrumentation will be the case study for adopting *UIActions.
  • You can now reference the Web team's approach to tracking the current clientPref sidebar as a point of reference while addressing tasks related to this User Story ticket.

To observe the client preferences event logging, please follow the steps outlined below:

  1. In Vector/skin.json, enable VectorClientPreferences to render the user preferences sidebar.
  2. Test the functionality of the radio buttons.
  3. Observe that the DesktopWebUIActions schema event, specifically event.action = click, is triggered.
  4. For more detailed information, please refer to T350417 (User Preferences Sidebar and Dropdown Panel Click Tracking Instrumentation)

Note: Final click-tracking spec is not yet available, but I've made a list below of events that are now being tracked in the sidebar. As a reminder, you can also refer to the Interface_Customization_Instrumentation_Spec for baseline instrumentation.

ActionEventValue
ClickUser chooses Column Width Standardvector-client-pref-vector-feature-limited-width-value-1
ClickUser chooses Column Width Widevector-client-pref-vector-feature-limited-width-value-0
ClickUser chooses Font Size Smallvector-client-pref-vector-feature-custom-font-size-value-0
ClickUser chooses Font Size Standardvector-client-pref-vector-feature-custom-font-size-value-1
ClickUser chooses Font Size Largevector-client-pref-vector-feature-custom-font-size-value-2
ClickUser chooses Font Size X-Largevector-client-pref-vector-feature-custom-font-size-value-3

Let me know if you need anything else from us in regards to our current client preferences instrumentation and analysis project.

To observe the client preferences event logging, please follow the steps outlined below:

  1. In Vector/skin.json, enable VectorClientPreferences to render the user preferences sidebar.
  2. Test the functionality of the radio buttons.
  3. Observe that the DesktopWebUIActions schema event, specifically event.action = click, is triggered.
  4. For more detailed information, please refer to T350417 (User Preferences Sidebar and Dropdown Panel Click Tracking Instrumentation)

A couple of things, I'll start with the minor first as it's fairly trivial and leads to the major:

Minor: It took me a while to figure out that this is currently only available for logged-in users (per https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/skins/Vector/+/refs/heads/master/resources/skins.vector.clientPreferences/clientPreferences.js#176).

Major: When I interact with a radio button, I see two identical events being submitted one after the other (with < 1 ms gap between submissions).

VirginiaPoundstone renamed this task from [User Story] Begin migrating *UIActions instrument to the Core Interaction API to [User Story] Partial migrating *UIActions instrument to the Core Interaction API.Nov 28 2023, 2:24 PM
VirginiaPoundstone renamed this task from [User Story] Partial migrating *UIActions instrument to the Core Interaction API to [User Story] Partial migration of *UIActions instrument to the Core Interaction API.Nov 28 2023, 2:35 PM

For the major issue, we have addressed it here.

Change 978718 had a related patch set uploaded (by Clare Ming; author: Clare Ming):

[schemas/event/secondary@master] Add custom schemas for *uiactionstracking instruments

https://gerrit.wikimedia.org/r/978718

@phuedx re: your example in the ticket description of mw.eventLog.submitInteraction():

'/analytics/product_metrics/web/ui_actions/1.0.0'

Should the custom schema belong in a sub-directory (product_metrics) inside where the web team keeps its own schemas (i.e. /analytics/mediawiki/product_metrics/ui_actions)?

I'm under the impression that we want to keep /analytics/product_metrics/web/ reserved for schemas that MP maintains (in the future this would contain say click or scroll or funnel etc directories) and then product teams would maintain their own custom schemas based off MP base schemas wherever they usually have their schemas contained -- inside a product_metrics subdirectory?

Also, I err'd on the side of keeping desktop + mobile separate because of the required properties. I noted on the patch that maybe they should be merged into a single custom schema but then we wouldn't be able to make the custom properties required for desktop and mobile respectively, and for data parity checks, would it be easier for the data analyst to compare the legacy schemas against their corresponding custom MP schemas vs a single custom schema that combines both?

Change 978947 had a related patch set uploaded (by Clare Ming; author: Clare Ming):

[operations/mediawiki-config@master] Add stream config for *uiactionstracking via Metrics Platform

https://gerrit.wikimedia.org/r/978947

Change 978948 had a related patch set uploaded (by Clare Ming; author: Clare Ming):

[mediawiki/extensions/WikimediaEvents@master] Update *uiactions instruments to send events via Metrics Platform

https://gerrit.wikimedia.org/r/978948

One more thing - not sure about the mobilewebuiactionstracking instrument - i'm guessing that client preferences are not applicable/available yet for the mobile view?

It's working to capture the init and other click events (see https://phabricator.wikimedia.org/F41548099#5980) but it's not readily apparent to me how client prefs work in the mobile view

@KSarabia-WMF - can you confirm if I should be able to set client prefs in mobile? If yes, I may need to pair with you on how to see/use that feature.

Testing Instructions

These testing instructions are OUTDATED - please refer to T351298#9373969 instead

  1. To see the event validate locally, pull:
  1. Add the following config to LocalSettings.php:
$wgEventStreams = [
	'mediawiki.web_ui_actions_desktop.migrated' => [
		'schema_title' => '/analytics/mediawiki/product_metrics/desktopwebuiactionstracking',
		'producers' => [
			'metrics_platform_client' => [
				'provide_values' => [
					'page_namespace_id',
					'performer_is_logged_in',
					'performer_session_id',
					'performer_pageview_id',
					'performer_edit_count_bucket',
					'mediawiki_skin',
					'mediawiki_database'
				],
			],
			'sample' => [
				'unit' => 'pageview',
				'rate' => 1,
			],
		],
	],
	'mediawiki.web_ui_actions_mobile.migrated' => [
		'schema_title' => '/analytics/mediawiki/product_metrics/mobilewebuiactionstracking',
		'producers' => [
			'metrics_platform_client' => [
				'provide_values' => [
					'page_namespace_id',
					'performer_is_logged_in',
					'performer_session_id',
					'performer_pageview_id',
					'performer_edit_count_bucket',
					'performer_groups',
					'mediawiki_skin',
					'mediawiki_database'
				],
				'curation' => [
					'mediawiki_skin' => [
						'in' => [ 'minerva' ],
					],
				],
			],
			'sample' => [
				'unit' => 'pageview',
				'rate' => 1,
			],
		],
	],
];

$wgEventLoggingStreamNames = array_keys( $wgEventStreams );
  1. Make sure you also have VectorClientPreferences set to true for logged_in in LocalSettings.php:
$wgVectorClientPreferences = [
	"logged_in" => true,
	"logged_out" => false,
];
  1. Start up your local Mediawiki instance and make sure to log in.
  1. Start up your local EventGate devserver:
(if using Docker)
docker compose up --force-recreate eventlogging
  1. Navigate to localhost in your browser - you should see the client preferences in the right sidebar:

Screenshot 2023-11-30 at 1.59.35 AM.png (1×2 px, 695 KB)

  1. Toggle the Text radio buttons in the browser and observe the event in your local EventGate:

Screenshot 2023-11-30 at 2.01.59 AM.png (1×3 px, 596 KB)

\o/

@KSarabia-WMF - can you confirm if I should be able to set client prefs in mobile? If yes, I may need to pair with you on how to see/use that feature.

@cjming Right now, the client preferences functionality is only in Vector skin, so only available on Desktop size/Desktop view.

@KSarabia-WMF @Jdlrobson some Qs for you both on how we can move forward with logging *uiactionstracking events via MP.

As you can see in the preliminary patch sets for this ticket, my initial thought was to separate the custom schemas like they are now in the legacy folder. But the desktop one only has 2 custom properties (the rest are covered by MP base web schema) is_sidebar_collapsed and viewport_size_bucket and the mobile one only has 1 custom property modes which after some discussion with @phuedx seems like it could be easily mapped to action_context in which case the mobile instrument wouldn't need an additional custom schema specifically for it.

We propose creating only one custom schema for both instruments and make the custom properties (is_sidebar_collapsed and viewport_size_bucket) not required.

For context, there is a MP common fragment that holds action-related properties and is included with the MP web base schema.

My questions are:

  1. Are you ok with having both instruments validate their events against a single custom schema that will only have is_sidebar_collapsed and viewport_size_bucket as its custom non-required properties?

    If so, there is a discrepancy between the 2 instruments -- one parameter in mobileWebUIActions.js's logEvent() called destination is not passed into the logEvent() function in desktopWebUIActions.js. From what we can tell, it's not being used anywhere but here. Are you ok with dropping this parameter for the mobile instrument when logging the event via MP if we combine schemas?

    Otherwise we could map destination to action_subtype which will only be populated in the mobile instrument, and not in the desktop one.
  2. In both instruments, we would map the name parameter to action_source - seem reasonable?

@cjming 1 and 2 sound very reasonable to me. I also spoke with Jennifer about the destination field and we're totally fine with dropping.

  1. Yes drop destination
  2. Sounds fine

I also talked to @phuedx about dropping skinVersion and mode. The former is just an alias for skin and the latter would be better served by a tag Advanced Mobile Contributions

thanks @KSarabia-WMF thanks @Jdlrobson for confirming

So I refactored patches to use one custom schema and one stream configuration for both instruments, and dropped destination from the mobile instrument.

@Jdlrobson re: modes -- Sam and I were talking about adding tagging to events - iirc he's going to write up a ticket for our backlog to add that feature. For now, I err'd on the side of mapping modes to action_context from the mobile instrument.

All 3 patches for review have been updated (cc @phuedx):

And new testing instructions below:


Testing Instructions

  1. To see the event validate locally, pull:
  1. Add the following stream config to LocalSettings.php:
$wgEventStreams = [
	'mediawiki.web_ui_actions' => [
		'schema_title' => '/analytics/mediawiki/product_metrics/web_ui_actions',
		'producers' => [
			'metrics_platform_client' => [
				'events' => [
					'webuiactions_log.',
					'click',
					'init',
				],
				'provide_values' => [
					'page_namespace_id',
					'performer_is_logged_in',
					'performer_session_id',
					'performer_pageview_id',
					'performer_edit_count_bucket',
					'performer_groups',
					'mediawiki_skin',
					'mediawiki_database'
				],
			],
			'sample' => [
				'unit' => 'pageview',
				'rate' => 1,
			],
		],
	],
];

$wgEventLoggingStreamNames = array_keys( $wgEventStreams );
  1. Make sure you also have VectorClientPreferences set to true for logged_in in LocalSettings.php:
$wgVectorClientPreferences = [
	"logged_in" => true,
	"logged_out" => false,
];
  1. Start up your local Mediawiki instance and make sure to log in.
  1. Start up your local EventGate devserver:
(if using Docker)
docker compose up --force-recreate eventlogging
  1. Navigate to localhost in your browser - you should see the client preferences in the right sidebar:

Screenshot 2023-11-30 at 1.59.35 AM.png (1×2 px, 695 KB)

  1. Note the init event when you first open the page in your browser:

Screenshot 2023-11-30 at 9.25.45 PM.png (1×3 px, 700 KB)

  1. Toggle the Text radio buttons in the browser and observe the event in your local EventGate:

Screenshot 2023-11-30 at 9.26.16 PM.png (1×3 px, 760 KB)

  1. Click on Mobile view link at the bottom of the footer and note the init event when you first switch to mobile view:

Screenshot 2023-11-30 at 9.27.12 PM.png (1×3 px, 670 KB)

  1. Click on the main menu hamburger icon and note the associated click event:

Screenshot 2023-11-30 at 9.27.34 PM.png (1×3 px, 679 KB)

  1. Click on an edit icon and note the associated click event:

Screenshot 2023-11-30 at 9.28.22 PM.png (1×3 px, 692 KB)

ʕ•́ᴥ•̀ʔっ

We've tested this and can confirm the events happening locally.

Init in local:

Screenshot 2023-12-01 at 12.23.29 PM.png (1×2 px, 303 KB)

Editing in local:

Screenshot 2023-12-01 at 12.36.34 PM.png (1×2 px, 373 KB)

Change 978718 merged by jenkins-bot:

[schemas/event/secondary@master] Add custom schema for *uiactionstracking instruments

https://gerrit.wikimedia.org/r/978718

Change 978948 merged by jenkins-bot:

[mediawiki/extensions/WikimediaEvents@master] Update *uiactions instruments to send events via Metrics Platform

https://gerrit.wikimedia.org/r/978948

Change 978947 merged by jenkins-bot:

[operations/mediawiki-config@master] Add stream config for *webuiactions via Metrics Platform

https://gerrit.wikimedia.org/r/978947

Mentioned in SAL (#wikimedia-operations) [2023-12-05T21:42:12Z] <jforrester@deploy2002> Started scap: Backport for [[gerrit:977785|Define the corresponding stream for scroll (T350883)]], [[gerrit:978947|Add stream config for *webuiactions via Metrics Platform (T351298)]]

Mentioned in SAL (#wikimedia-operations) [2023-12-05T21:43:50Z] <jforrester@deploy2002> ksarabia and jforrester and cjming: Backport for [[gerrit:977785|Define the corresponding stream for scroll (T350883)]], [[gerrit:978947|Add stream config for *webuiactions via Metrics Platform (T351298)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)

Mentioned in SAL (#wikimedia-operations) [2023-12-05T22:01:14Z] <jforrester@deploy2002> Finished scap: Backport for [[gerrit:977785|Define the corresponding stream for scroll (T350883)]], [[gerrit:978947|Add stream config for *webuiactions via Metrics Platform (T351298)]] (duration: 19m 01s)