Lets track the changes we see here for FF57 in this tasks so it's easier for Mozillians (and others) to follow what we do.
Possible problems in Navigation Timings?
I've seen that if we don't clear the cache between runs, backend times like connectEnd, connectStart, responseEnd , requestStart and responseStart is wrong. When I automate and clear the cache between runs the metrics is correct. If I use an old Firefox it works. It could either be something with Selenium/Gecko/Firefox 57 or it could be something that also affect real users. The thing is that video and VisualMetrics looks ok between runs (but navigation timings is wrong) so maybe this can affect our RUM metrics (or do we just disregard them). You can see the faulty metrics here: https://results.sitespeed.io/en.wikipedia.org/2017-11-14-22-34-52/pages/en.wikipedia.org/wiki/Barack_Obama/3.html#browsertime
Reported at https://bugzilla.mozilla.org/show_bug.cgi?id=1417308
Comparing FF 54, 57 and latest Chrome
https://dashboard.sitespeed.io/dashboard/db/browsers
Running on AWS, c4.large Linux, connectivity type cable, taking the median of 7 runs (30fps). Looks like on the Facebook page we have regression in some values. Else 57 is faster than 54.
Also the metrics seems more unstable for 57 than 54 and metrics. Also Chrome is actually running with trace-logging turned on (adding x ms to all metrics).
Here's the full result of all tested pages for the history.
Comparing FF56 and 57
Running WebPageTest, c4.large windows, 10 fps, cable connectivity, 5 runs. FF 57 was picked up something like 2017-11-14 17.00 UTC.
It seems like fully loaded is better in 57 across the board. For our Facebook page we can also here see a regression (WPT picked up 57 late the 14th):
Fully loaded looks better but we have high measurements that we haven't seen before:
It also looks like reported TTFB increased (but that I guess it can depend on how WPT measure it):
For Sweden we can see better metrics on SpeedIndex & first visual change but the big change happened there before 57 was released:
