Page MenuHomePhabricator

Review Catchpoint's Speed Index feature
Closed, ResolvedPublic

Description

Catchpoint released Speed Index as a new feature:

New Metric: Speed Index
NEW FEATURE
Cortex -- Catchpoint's March 2017 release -- brought a new metric: Speed Index. Speed Index is a measure of visual completeness on the page, giving you greater insight into what users actually see when interacting with your web applications.

Speed Index is best used in conjunction with timing metrics such as Document Complete and Render Start. While those metrics are still important to measure, and will remain so, Speed Index is a new, valuable way to gauge the user experience.

Note: Speed Index is available for Catchpoint Web and Transaction tests running on the Chrome monitor. In order to capture Speed Index, the additional setting Capture Filmstrip must be enabled for the test.

We should check if we're billed for usage or if it we can use it "as much as we like" under our existing Catchpoint subscription.

Related Objects

Event Timeline

I think when I tried it out the number of "points" increased (yep they have point system, so you pay points by money). But I don't know what kind of deal we have? We also make one run per test if I remember correctly.

It would be nice to have better control of what we run, right now we only have one full test (every 8th hour) that do login. And couple of more running Chrome but probably could be just a GET.

Also would be interesting to know more about their implementation, it's Chrome only right? Then they are probably using SpeedLine or something similar behind the scene. I haven't seen any case of collecting Chrome tracelogs that doesn't impact the metrics in Chrome, the only way today to do it right is to record a video.

SpeedLine does its Speed Index calculation based on screenshots, though, doesn't it? It must be a visual thing anyway, since they optionally offer the filmstrip. Would that require tracelogs?

yep the screenshots are collected in the Chrome tracelog with disabled-by-default-devtools.screenshot so it probably would affect the other metrics, but I haven't done a study. But for example turning on only collect the default timeline tracelog for Chrome affects the metrics on WPT.

However since neither Catchpoint/Dynatrace (also releases SpeedIndex a while ago) shows open how they do things, I'm only guessing.

Krinkle triaged this task as Medium priority.Apr 12 2017, 7:55 PM
Krinkle added a subscriber: Krinkle.

While we might use Catchpoint in the future for testing from various geo locations, but at the moment we don't use it, so we should at least remove our old tests to save points.

Let's make a list of the existing tests we have configured in Catchpoint and triage which are can be removed if they're not claimed by any other team.

In our team meeting today @Peter also mentioned that many of the tests from ops can be done in a more efficient (and cost-effective way) by using a cheaper browsers instead of Chrome (e.g. PhantomJS) for tests that are mainly intended to do uptime and latency checks from different locations.

We actually have only one performance test, as far as I can see. It uses Chrome, which makes it "expensive", but it's justified since it logs in. But there indeed seem to be significant savings to be had on the availability tests ops has set up.

Gilles claimed this task.

We tried it for the Asia metrics and it sucked, huge variations between runs, completely unusable.