I think when I tried it out the number of "points" increased (yep they have point system, so you pay points by money). But I don't know what kind of deal we have? We also make one run per test if I remember correctly.

It would be nice to have better control of what we run, right now we only have one full test (every 8th hour) that do login. And couple of more running Chrome but probably could be just a GET.

Also would be interesting to know more about their implementation, it's Chrome only right? Then they are probably using SpeedLine or something similar behind the scene. I haven't seen any case of collecting Chrome tracelogs that doesn't impact the metrics in Chrome, the only way today to do it right is to record a video.

SpeedLine does its Speed Index calculation based on screenshots, though, doesn't it? It must be a visual thing anyway, since they optionally offer the filmstrip. Would that require tracelogs?

yep the screenshots are collected in the Chrome tracelog with disabled-by-default-devtools.screenshot so it probably would affect the other metrics, but I haven't done a study. But for example turning on only collect the default timeline tracelog for Chrome affects the metrics on WPT.

However since neither Catchpoint/Dynatrace (also releases SpeedIndex a while ago) shows open how they do things, I'm only guessing.

We actually have only one performance test, as far as I can see. It uses Chrome, which makes it "expensive", but it's justified since it logs in. But there indeed seem to be significant savings to be had on the availability tests ops has set up.