Page MenuHomePhabricator

Sticky Header: Create schema to track returning to the top of the page
Closed, ResolvedPublic5 Estimated Story Points

Description

Background

Based on T287709: [SPIKE] Explore instrumentation for sticky header work and subsequent discussion, we will be creating a schema that tracks whether the user is visiting the top of the page. This schema will be used to answer the research questions outlines in T287709: [SPIKE] Explore instrumentation for sticky header work

Acceptance criteria

  • Should log the user session ID
  • Should log the page id where the event happens
  • Should log whether user is anonymous
  • Should be sampled and configurable, by default 0 to allow us to roll out as needed.
  • Should only run on the Vector skin
  • An init event should be logged that a user is eligible for "scroll to top events"
  • An event should be logged when the user scrolls to the top (See definition)
  • An event should be logged when a user is believed to have scrolled to the top. For now, this is defined as when the following sequence of events occurs:
  • The sticky header displays
  • At least 5 seconds pass
  • The sticky header disappears (because the user scrolls to to the top)
  • The code should live in WikimediaEvents.Use mw.track in Vector if you need to trigger events relating to codepoints in Vector.

Developer notes

sticky header instrumentation spec

QA steps

  • For testing event logging locally, the following config will be needed in local settings:
$wgVectorWebABTestEnrollment = [
	'name' => 'sticky_header_2021_11',
	'enabled' => true,
	'buckets' => [
		'unsampled' => [
			'samplingRate' => 0.1,
		],
		'control' => [
			'samplingRate' => 0.3,
		],
		'stickyHeaderDisabled' => [
			'samplingRate' => 0.3,
		],
		'stickyHeaderEnabled' => [
			'samplingRate' => 0.3,
		]
	]
];
  • The following config should also be updated locally for testing purposes only in ../extensions/WikimediaEvents/extension.json:
		"WMEWebUIScrollTrackingSamplingRate": {
			"value": 1
		},
		"WMEWebUIScrollTrackingTimeToWaitBeforeScrollUp": {
			"value": 5000
		},
  • Create some new test users to make sure you have one in each group: control, stickyHeaderDisabled, stickyHeaderEnabled (eventlogging will indicate which bucket the user is in by the group key if testing T292587 alongside this ticket -- or inject debugger code in L524 of stickyHeader.js to view which group the user is assigned).

debugger injected into javascript if using:

// eslint-disable-next-line no-debugger
debugger;
  • For all groups, the scroll-to-top event should be logged every time after one has scrolled down, pauses for 5 seconds, and then scrolls back up to the top of the page (i.e. above the page title).

Screen Shot 2021-11-01 at 2.32.59 PM.png (808×2 px, 176 KB)

QA Results - Beta

ACStatusDetails
1T292586#7556542

QA Results - Prod

ACStatusDetails
1T292586#7573091

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Change 731156 abandoned by Clare Ming:

[schemas/event/secondary@master] Add new schema for desktop UI scroll tracking.

Reason:

not sure what happened here but couldn't get git hooks to run so created new patch for review

please test/review https://gerrit.wikimedia.org/r/c/schemas/event/secondary/+/735003 instead

https://gerrit.wikimedia.org/r/731156

hi @ovasileva @jwang quick Q about logging the page id for the scroll-to-top event -- currently this is captured by the web_pageview_id property in the web_identifiers fragment.

From the documentation there:

This identifier is randomly generated the first time it is requested by the instrumentation code on any page view and persists for the lifetime of the page. When the user navigates to another page or refreshes/reloads the page, this identifier disappears and a new one is regenerated (when needed).

Is this ok? i.e. should the page id persist between page reloads/refreshes?

Is this ok? i.e. should the page id persist between page reloads/refreshes?

Sorry to chime in with unsolicited advice, hopefully that's okay. Rather than preserving the per-page-view random ID across reloads/refreshes may I suggest just including the MediaWiki page ID as a field in the schema (with the value being mw.config.get('wgArticleId'))?

Agree with @mpopov, we need consistent page ID across refreshes. MediaWiki page ID is the one which serves the purpose.

thanks @mpopov @jwang

@jwang can you confirm if page id is required? is pageview id (token) sufficient if we're just tracking how many times the user scrolled to the top?

Hi @cjming, it depends on whether we need breakdown dimension by page in analysis. We don't know how the user behavior would look like because we don't have data yet. My hypothesis is that the user scroll behavior might be correlated with page character (content page or talk page, long page or short page, etc) . page_id ( article ID) provided a join key to connect the scroll schema with page schema, which contains page character info, like page_namespace, page_len. page_id ( article ID) also provided a dimension to compare scroll behavior on the same page (like main page) across AB groups and cross sessions.

web_pageview_id (token) doesn't provide any additional info about page itself. If we just need a simple analysis, it might be enough.

Above are considerations from analyst's point of view. Engineer and PM @ovasileva can make the call based on the needs and efforts.

Hi @cjming, it depends on whether we need breakdown dimension by page in analysis. We don't know how the user behavior would look like because we don't have data yet. My hypothesis is that the user scroll behavior might be correlated with page character (content page or talk page, long page or short page, etc) . page_id ( article ID) provided a join key to connect the scroll schema with page schema, which contains page character info, like page_namespace, page_len. page_id ( article ID) also provided a dimension to compare scroll behavior on the same page (like main page) across AB groups and cross sessions.

Looking across pages of different length (or being able to filter for longer pages only) would be valuable data to have, although not highest priority. If it's fairly straightforward, I'd say let's go ahead and add it. If there's significant challenges we can reconsider. Will make a note to discuss this in standup today

cjming assigned this task to nray.

Change 735003 merged by jenkins-bot:

[schemas/event/secondary@master] Add new schema for web UI scroll tracking.

https://gerrit.wikimedia.org/r/735003

cjming removed cjming as the assignee of this task.Nov 3 2021, 2:32 AM
cjming assigned this task to nray.

Change 731157 merged by jenkins-bot:

[mediawiki/extensions/WikimediaEvents@master] Add scroll event logging

https://gerrit.wikimedia.org/r/731157

Change 737497 had a related patch set uploaded (by Clare Ming; author: Clare Ming):

[mediawiki/extensions/WikimediaEvents@master] Update web UI scroll js

https://gerrit.wikimedia.org/r/737497

Change 737497 merged by jenkins-bot:

[mediawiki/extensions/WikimediaEvents@master] Update web UI scroll js

https://gerrit.wikimedia.org/r/737497

Change 734778 merged by jenkins-bot:

[mediawiki/skins/Vector@master] Add scroll event + init A/B test logging to sticky header, AB js

https://gerrit.wikimedia.org/r/734778

Change 742524 had a related patch set uploaded (by Clare Ming; author: Clare Ming):

[operations/mediawiki-config@master] Enable scroll tracking for all users

https://gerrit.wikimedia.org/r/742524

Change 742524 merged by jenkins-bot:

[operations/mediawiki-config@master] Enable scroll tracking for all users

https://gerrit.wikimedia.org/r/742524

Mentioned in SAL (#wikimedia-operations) [2021-11-30T00:17:55Z] <catrope@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:742524|Enable scroll tracking for all users (T292586)]] (duration: 00m 55s)

Thanks to @cjming change above we should now be able to QA this in production and beta cluster (hopefully) prior to launch.

Change 742834 had a related patch set uploaded (by Clare Ming; author: Clare Ming):

[operations/mediawiki-config@master] Add mediawiki.web_ui_scroll stream

https://gerrit.wikimedia.org/r/742834

Thanks to @cjming change above we should now be able to QA this in production and beta cluster (hopefully) prior to launch.

Great news! @cjming, @Jdlrobson - how should we approach QA for this one? Should we hand it over directly to @jwang/QA as a part of T294639: Schema QA sticky header instrumentation?

^ We need to backport the above change today first. Once that's done we'll check events are showing up in https://grafana.wikimedia.org/d/000000566/overview?viewPanel=29&orgId=1 at which point we can assign QA to Jennifer

Change 742834 merged by jenkins-bot:

[operations/mediawiki-config@master] Add mediawiki.web_ui_scroll stream

https://gerrit.wikimedia.org/r/742834

Mentioned in SAL (#wikimedia-operations) [2021-12-01T19:11:33Z] <taavi@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:742834|Add mediawiki.web_ui_scroll stream (T292586)]] (duration: 00m 57s)

hi @ovasileva -- backport is done and data is showing up in grafana so I'll go ahead and assign this to @jwang for QA

Change 743037 had a related patch set uploaded (by Clare Ming; author: Clare Ming):

[mediawiki/skins/Vector@master] Update scroll observer to allow event logging

https://gerrit.wikimedia.org/r/743037

Change 743013 had a related patch set uploaded (by Jdlrobson; author: Clare Ming):

[mediawiki/skins/Vector@wmf/1.38.0-wmf.9] Update scroll observer to allow event logging

https://gerrit.wikimedia.org/r/743013

Change 743037 merged by jenkins-bot:

[mediawiki/skins/Vector@master] Update scroll observer to allow event logging

https://gerrit.wikimedia.org/r/743037

Change 743013 merged by jenkins-bot:

[mediawiki/skins/Vector@wmf/1.38.0-wmf.9] Update scroll observer to allow event logging

https://gerrit.wikimedia.org/r/743013

Mentioned in SAL (#wikimedia-operations) [2021-12-02T00:24:41Z] <urbanecm@deploy1002> Synchronized php-1.38.0-wmf.9/skins/Vector/: a7586cd4a2559248ea1fd29cf74de535de016501: Update scroll observer to allow event logging (T292586) (duration: 00m 57s)

Hi everybody,

I reached this task after seeing https://phabricator.wikimedia.org/rOMWCab7c3a1b5c01e85cb93b625334500eb4a51e64d3, the mediawiki.web_ui_scroll scheme in Eventgate is failing validations consistently:

Grafana link
logstash dashboard

The majority of the errors are related to '' should have required property 'access_method'.

@elukey it sounds like some more work might be needed on T294246.
We can always disable this as a temporary measure.

thanks @elukey for sounding the alarm -- just backported a fix and it looks like the errors are dropping - sorry about that!

Test Result - Beta

Status: ✅ Pass
Environment: beta
OS: macOS Monterey
Browser: Chrome
Device: MBP
Emulated Device:NA

Test Artifact(s):

QA Steps

✅ AC1: For all groups, the scroll-to-top event should be logged every time after one has scrolled down, pauses for 5 seconds, and then scrolls back up to the top of the page (i.e. above the page title).

@cjming, I tried for a couple of users, and I get to scroll-to-top event, but I cannot see the groups. Let me know if there's a way to find the groups in beta (and eventually prod) or if this is sufficient.

This is a pass per T292586#7556886

Screen Shot 2021-12-08 at 6.14.30 AM.png (235×675 px, 43 KB)

hi @Edtadros - the group you already QA'd in T292587 -- sorry for the confusion. If you were testing this ticket alongside the A/B test enrollment ticket (T292587), you would see both the scroll-to-top event logged and the A/B test enrollment event logged one after the other in your local event stream. The latter has the group. Imho I think the testing you've already done is sufficient and covers the ACs here and in T292587.

This comment was removed by Edtadros.

Test Result - Prod

Status: ✅ Pass
Environment: testwiki
OS: macOS Monterey
Browser: Chrome
Device: MBP
Emulated Device:NA

Test Artifact(s):

QA Steps

✅ AC1: For all groups, the scroll-to-top event should be logged every time after one has scrolled down, pauses for 5 seconds, and then scrolls back up to the top of the page (i.e. above the page title).

Screen Shot 2021-12-15 at 9.33.46 AM.png (891×871 px, 196 KB)

Hi, I find that 2 data fields in mediawiki_web_ui_scroll schema are not in original sticky header instrumentation spec. Can anyone help me to understand what they are tracking?

Fieldexample data
meta.id80fe8b7b-0d4c-45b8-9c70-66ab6cbb905
meta.request_id9068b14b-85d6-4848-b09f-ba4d7b94d62

hi @jwang - the 2 properties meta.id + meta.request_id come with the common fragment that is included in the web_ui_scroll schema.

Per Common 2.0.0 fragment schema:

meta:
  ...
  properties:
    ...
    id:
      description: Unique ID of this event
      type: string
    request_id:
      description: Unique ID of the request that caused the event
      type: string
    ...

Hi @cjming, thanks for the info. Then what is the request referring to in the common fragment?

Then what is the request referring to in the common fragment?

In general it depends. This request_id is used for request tracing. For example, if this event were generated by MediaWiki server side, it would be the X-Request-ID header set by MediaWiki. EventGate will create a new one for this if it isn't already set.

Basically, if all systems respect this and forward it properly, it is possible to reconstruct a 'what triggers what' series of requests and events in logs.

See also:

In general it depends. This request_id is used for request tracing. For example, if this event were generated by MediaWiki server side, it would be the X-Request-ID header set by MediaWiki. EventGate will create a new one for this if it isn't already set.
Basically, if all systems respect this and forward it properly, it is possible to reconstruct a 'what triggers what' series of requests and events in logs.

@Ottomata, thank you very much for the explanation!

Hi, I explored the mediawiki_web_ui_scroll schema. It seems that the data in dt field is not right. It records some dates in 2008 when this schema is not enabled. It also records many dates in 2022, which is the future. It doesn't match with the partition year, month day.
This data issue shows for all wikis not just on particular one wiki. Is it a known issue?

Here are a few examples:

date_timesubstr_dtyearmonthdayeventssessionspages
2008-12-312008-12-3120211213111
2020-12-072020-12-072021127111
2022-01-132022-01-1320211213111
2022-01-212022-01-2120211215312
2022-01-222022-01-2220211213111
2021-12-152021-12-152021124 444
2021-12-152021-12-1520211213231717
2021-12-152021-12-1520211216 1408789

Query Code

SELECT to_date(dt) AS date_time, substr(dt,1,10),year, month,day, COUNT(1) AS events,
COUNT(DISTINCT web_session_id) AS sessions,
COUNT(DISTINCT page_id) AS pages
FROM event.mediawiki_web_ui_scroll
WHERE year=2021
GROUP BY to_date(dt),substr(dt,1,10),year, month,day

Hi, I explored the mediawiki_web_ui_scroll schema. It seems that the data in dt field is not right. It records some dates in 2008 when this schema is not enabled. It also records many dates in 2022, which is the future. It doesn't match with the partition year, month day.
This data issue shows for all wikis not just on particular one wiki. Is it a known issue?

Here are a few examples:

date_timesubstr_dtyearmonthdayeventssessionspages
2008-12-312008-12-3120211213111
2020-12-072020-12-072021127111
2022-01-132022-01-1320211213111
2022-01-212022-01-2120211215312
2022-01-222022-01-2220211213111
2021-12-152021-12-152021124 444
2021-12-152021-12-1520211213231717
2021-12-152021-12-1520211216 1408789

Query Code

SELECT to_date(dt) AS date_time, substr(dt,1,10),year, month,day, COUNT(1) AS events,
COUNT(DISTINCT web_session_id) AS sessions,
COUNT(DISTINCT page_id) AS pages
FROM event.mediawiki_web_ui_scroll
WHERE year=2021
GROUP BY to_date(dt),substr(dt,1,10),year, month,day

@Ottomata - we are noticing this across two different schemas. Do you know if there's an issue with the new event platform?

@jwang @ovasileva Andrew is the expert, but I believe this is happening because dt has switched to meaning the time according to the client. So if someone, say, has the time on their phone set to 2008 or 2022, dt would reflect that. On the other hand, meta.dt (which is used to set the partition fields) is the time our server received the event, which would still be 2021.

@nshahquinn-wmf, thank you very much for the info! Then it's not a bug.

@nshahquinn-wmf, thank you very much for the info! Then it's not a bug.

Sounds good, in that case @jwang - I'll go ahead and resolve this one