Let me just fix that.

This was fixed in T215160

I need your input on this @aaron @Krinkle and @Gilles before I start to collect metrics. Am I doing it correct? Anything I should change?

I've started with today. I've added tests running from Sweden and Mumbai as a start. My plan is like: If everything looks ok I'll start by documenting the setup tomorrow, and the Thursday close down the test and collect and so a summary of the result. Lets do a full run down at Wikitech, so we have the data for the future and then do a more light weight blog post about it.

This looks good now. Mobile changed like this for all URLs I checked:

Looks much better!

I did some more changes and now uses 0.05% diff that's ok, that made it work so it runs in the servers now. Let me have a look tomorrow that everything is ok and then close.

No that didn't help, there is still something like 13 s last visual change happening for Obama on mobile. I've changed so that we look a 0.01% of the pixels can differ (that worked locally) but it seems that the diff is too small. Let me increase it a little and try again tomorrow.

I've pushed the fix now (with an annotation), I'll check that it really works before I close it.

There's two ways to fix this: when visual metrics removes frames from the end, it uses a fuzz factor of 10% (that's default that is used all over) and then compares pixel by pixel and the there must be zero pixels diff, for making sure that the screenshots are the same. However when looking for Visual Elements, the element also uses a 10 % fuzz but a 2% of pixels that can differs.

Adding an example file. Running default visual change gives last visual change something like 11 seconds but the "real" one should be more like 6 seconds.

Hmm this was really annoying. I've been trying to backtrack/understand how this works. I've been trying with two images that is generated with a diff with VisualMetrics:

Let me know if you need any help with it!

Cool, thanks @Niedzielski FYI I added one annotation to that dashboard:

If its ok to make it possible to disable it, then do it.

Enabling them by setting:

I've added them here: https://wikitech.wikimedia.org/wiki/Performance/Metrics#Synthetic_metrics

Upgraded to c5.xlarge. There's an annotation in Grafana:

Had a go today and I could go through the full flow. Now we just need to decide what to measure :)

This is cleaned up now.

Updated all the alerts.

Argh that didn't kick in, but manually removing old entries work, so I'll do that for the three beta mobile URLs so we can easily get to the interesting runs and then check again on Monday that the rest is ok.

Ok, I've made a mistake when I setup the delete rules on S3 and only matched them to a couple of the folders, so beta hasn't been deleted since I added it. I've added rules for all folders and hopefully this will self-heal when older ones are automatically deleted so I'll have a look first thing tomorrow.

I've activated them again for authenticated in Chrome, it looks good now at least, taking avg over a large time span and we can find a 40 ms change.

Tested now and it seems that subfolders isn't supported (I couldn't find in the GUI how to do it) and no one else has done any. No mentioning about it http://docs.grafana.org/reference/dashboard_folders/

I've changed on WebPageReplay desktop/WebPageReplay mobile and WebPageTest to alert on missing metrics.

I've changed to If no data or all values are null SET STATE TO ALERT on https://grafana.wikimedia.org/d/000000490/webpagereplay-mobile-alerts?orgId=1 first visual change and Speed Index and let try that for a while.

This is running all servers now, it looks ok. Some small changes in some SpeedIndex and FirstVisual change and I think the reason is that the default view port on desktop has been changed in this release.

I've pushed this on the Firefox instance and let me verify that it works there before I push it on the other instances.

As I understands it, the only way to install is to have a full package file (apk) but that are not available for download for Chrome.

Number 1 will be made configurable (introduced 7 months ago!). Number 2 has been reverted.

I fixed it now by doing some code changes on our local instance.

Argh this morning the start test URL is also failing: http://wpt.wmftest.org/testlog/1/

I've updated the code to the latest version but no luck. I'll dig through the logs tomorrow.

Hmm trying manually a couple of days work http://wpt.wmftest.org/testlog.php?days=4&filter=&all=on but http://wpt.wmftest.org/testlog.php?days=30&filter=&all=on do not.

Let me try this out to see what kind of regression % we can catch.

We don't have a good way to do to add a lot of alerts (since templating doesn't work in Grafana). That means that to setup alerts for a new Wikipedia we probably should create a new dashboard so we don't overlap the current one. Let us wait for a while and see which is the best way forward.

So I spent a lot of time on this with no luck. The problem at the moment is the ARM structure, that means we need to have a completely other Docker setup (installing versions compiled for ARM), and that will add overhead that I don't want at the moment. Also Chrome and Chromedriver do not have any official ARM builds (but that may change since Edge moves to Chromium and Windows uses ARM). Let close this for now and take it up again if we get a Chrome setup that automatically works on ARM.

Added compare to last week graphs (scroll down):
https://grafana.wikimedia.org/d/000000572/webpagereplay-drilldown?orgId=1

I'll like folders because it makes things more accessible for new users that comes to our dashboards. I've just started by adding a Performance dashboard and moved the WebPageTest/WebPageReplay dashboards there.

All WebPageReplay dashboards are now updated.

Added info boxes with definition of each metrics that makes it easier to understand what we are showing in the graphs.

I've cleaned up all old WebPageTest data (WebPageRelay most where ok). Two things: we have tested from different locations (most when we tried out the new Linux versions) and there must have been a glitch somewhere where metrics keys where sent in lower case only, created double metrics. All these are fixed now.

I've updated https://grafana.wikimedia.org/d/000000095/webpagetest-drilldown and it looks good for enwiki and enwiki-mobile. Let me have a look for the rest and when they are cleaned up we can close this!

And all URLs have been from these locations that we should not use:

• eu-central-1
• us-east-1-hosted-linux
• us-east-1-linux

I cannot reproduce now, seems to be fixed, thank you @CDanis and @fgiunchedi !

For browsertime we had one on the second key that was wrong *mobile* removed that too.

We currently have these for WebPageTest in Graphite as second key:

No couldn't get reverse tethering to work on those phones. But I don't think we need that.

This seems to work fine.