Multiple times we have collected wrong firstVisualChange from the video:
- T163199 - Password bubble and loading bar
- T135482 - Loading bar
- T129735 - First view content messed up second view
This happen again and it's hard to automate it in WebPageTest or whatever tool we use to calculate firstVisualChange. It has been changes in Chrome that has caused the problem some of the times.
There's two ways to tackle this:
- Make easier to check the screenshots of the runs. Now you need to go to our test instance, find the test, click on the test, go to one run and then choose filmstrip view. We could extract the images and have one page that shows the latest X runs filmstrips and we could easily verify them.
- We could setup an alert that alerts if the metrics are too unrealistic fast. Need to think about alert values but should be possible by looking at what kind of value we have seen before.