Page MenuHomePhabricator

[Session Length] Evaluate session length distribution for newer vs. older browsers and remove the supportsPassive check
Closed, ResolvedPublic

Description

The session tick instrument currently includes a test subfield indicating whether the browser supports passive event listeners. As discussed on T261701, the purpose of this collection is to determine what percentage of browsers do not support passive event listeners and whether there exists a difference in the distribution of session length results between newer and older browsers (i.e., between browsers that support passive event listeners and those that do not). If a small enough percentage of browers do not support passive event listeners or if no substantial difference exists in the results between the two groups, we can stop collecting session length data from these older browsers in order to spare them the performance penalty of doing so.

See also
https://www.mediawiki.org/wiki/Compatibility#Browsers
https://caniuse.com/passive-event-listener

Event Timeline

Looking at the data collected yesterday, 2/8/21, on the group 0 and 1 wikis:

  • Only 20,569 (0.77%) out of 2,656,586 initial ticks (tick = 0) were from browsers that did not support passive event listeners.
  • That said, the initial dropoff between ticks 0 and 1 was much sharper for the group that did not support passive event listeners than for the population as a whole. Only 10,138 (49.29%) of sessions in browsers not supporting passive event listeners made it to tick 1, vs. 1,993,578 (75.04%) of all sessions. After tick 1, the dropoff rate between subsequent ticks appears roughly similar.

Change 663291 had a related patch set uploaded (by Jason Linehan; owner: Jason Linehan):
[mediawiki/extensions/WikimediaEvents@master] Disable sessionTick on browsers w/o passive event listener support

https://gerrit.wikimedia.org/r/663291

Change 663291 merged by jenkins-bot:
[mediawiki/extensions/WikimediaEvents@master] Disable sessionTick on browsers w/o passive event listener support

https://gerrit.wikimedia.org/r/663291

kzimmerman moved this task from Sign-off to Doing on the Better Use Of Data board.

This change was based on a decision in the Better Use of Data check-in; we decided to disable session tick because it causes performance issues for older browsers. We are deactivating the supportsPassive check for now but may explore options in the future to collect data from older browsers.

@jlinehan can you add a note about passive event listener support to the Session Length documentation (under caveats: https://wikitech.wikimedia.org/wiki/Analytics/Data_Lake/Traffic/SessionLength#Caveats)? Once that's done you can move this back to signoff and I'll close as resolved.

Whoops, moving back to signoff for now.

Thanks @Mholloway ! Some followups:

The data collected showed that fewer than 5% of clients lack passive event listener support

On T274264#6816810, you wrote 0.77% ; would it be reasonable to say "fewer than 1%"?

and a cursory examination showed no meaningful difference in their distributions.

Suggested revision (again based on your earlier comment):

Browsers that don't support passive event listeners are less likely to progress beyond tick 0 (50% continue past tick 0, compared to 75% for all sessions). After tick 1, the drop-off rate between subsequent ticks appears roughly similar to all browsers.

(I think that's a meaningful difference, although we decided at this time that collecting the data from old browsers is not worth risking a negative user experience based.)

Thanks @Mholloway ! Some followups:

The data collected showed that fewer than 5% of clients lack passive event listener support

On T274264#6816810, you wrote 0.77% ; would it be reasonable to say "fewer than 1%"?

Yes, better.

and a cursory examination showed no meaningful difference in their distributions.

Suggested revision (again based on your earlier comment):

Browsers that don't support passive event listeners are less likely to progress beyond tick 0 (50% continue past tick 0, compared to 75% for all sessions). After tick 1, the drop-off rate between subsequent ticks appears roughly similar to all browsers.

(I think that's a meaningful difference, although we decided at this time that collecting the data from old browsers is not worth risking a negative user experience based.)

Yep, makes sense to me!

@kzimmerman Updated the wiki; please let me know if that looks good!

@Mholloway Looks good, closing this as resolved - thank you!