Study effect of proposed ResourceLoader feature test in production
Closed, ResolvedPublic

Description

The feature test proposed in https://gerrit.wikimedia.org/r/218955 (T102318) should be tested in production by running a sampled experiment that executes the proposed feature test and reports back to EventLogging whether it passed or not.

This will help gain confidence that this is an accurate reflection of our current status quo and a stable reference point from which we can move further.

If the total percentage of traffic passing the feature test (e.g. after a week) is below a certain threshold we can look at the misses and identify browsers we may be cutting out unknowingly. Then we'll decide whether we need to support them, or to document them as no longer supporting Grade A (since they'll naturally get the Grade C experience).

Krinkle created this task.Mar 4 2016, 11:41 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 4 2016, 11:41 PM
ori moved this task from Inbox to Next-up on the Performance-Team board.Mar 7 2016, 7:35 PM
Krinkle claimed this task.
Krinkle moved this task from Next-up to Doing on the Performance-Team board.Mar 28 2016, 7:18 PM

Change 280372 had a related patch set uploaded (by Krinkle):
Track ResourceLoader feature-test pass/fail in statsd

https://gerrit.wikimedia.org/r/280372

ori added a comment.Apr 5 2016, 12:55 AM

What should the "certain threshold" be for deciding whether or not to go with this change? And what should the thresholds for grade C or grade X be? It'd be better to decide that in advance. Looking at the data before setting thresholds will introduce bias.

Krinkle updated the task description. (Show Details)Apr 5 2016, 2:02 AM

I've updated the task description at T102318 with preamble, reasons and data about current Grade A/C/X ratio in production traffic.

Grade A+X browsers: 95.4% of pageview traffic (86.27% + 9.16%)
Grade C browsers: 4.85% of pageview traffic

The data collected for this task will only come from Grade A and X browsers.

All of the currently supported Grade A browsers passed the feature test in isolated testing. It'll be good to confirm this in the wild though (which will account for fringe browser plugins and obscure browser configuration options that may break, clobber or otherwise disable functionality).

Browsers that won't pass the feature test already suffer from partial loss of functionality and slower performance due them being unknown to us (and not listed in the Grade C blacklist).

Of the current 86.27% Grade A browsers, at least 99.9% should pass the feature test.

The impact on Grade X browsers is interesting to note, but can't be given any meaningful threshold. If it turns out that most of the Grade X browsers are not feature compatible with Grade A, then them being given Grade C experience going forward would be an improvement.

Change 280372 merged by jenkins-bot:
Collect impact of proposed ResourceLoader feature-test in statsd

https://gerrit.wikimedia.org/r/280372

Change 282744 had a related patch set uploaded (by Krinkle):
Collect impact of proposed ResourceLoader feature-test in statsd

https://gerrit.wikimedia.org/r/282744

Change 282744 merged by jenkins-bot:
Collect impact of proposed ResourceLoader feature-test in statsd

https://gerrit.wikimedia.org/r/282744

Krinkle closed this task as Resolved.Apr 14 2016, 7:25 PM

Change 295534 had a related patch set uploaded (by Krinkle):
Remove rlfeature data gathering

https://gerrit.wikimedia.org/r/295534

Change 295534 merged by jenkins-bot:
Remove rlfeature data gathering

https://gerrit.wikimedia.org/r/295534