This is done now, I'll add some graphs later on,

Sometimes I feel I miss a more high level version of the Chrome changelog: https://chromium.googlesource.com/chromium/src/+log/69.0.3497.100..70.0.3538.67?pretty=fuller&n=10000

Did you see https://tech.trivago.com/2018/10/12/building-fast-and-reliable-web-applications/ @Krinkle

Moved the last one to c5.large (and updated the docs) just now.

5.3.0 has been released.

I've updated the Firefox server to c5.xlarge and changed ewiki to also run on c5 series (c5.large). That one is also cheaper and a little bit faster.

Enabled them just now. 40 ms/40 points for Speed Index for all tests for now.

This is what it looks like going back 90 days:

It is really one page that has the problem: The facebook page. I can see that the difference between runs are that sometime we spend 100ms+ more on UpdateLayoutTree.

Woho! Templates in Grafana was released in 5.3.0 beta 2 (there's a beta 3 out there now). This means that when we upgrade to 5.3.0 we could potentially start using the Grafana API.

Let us focus on:

Let us do T195233 instead

I've added a script that stores all error logs (no so far) in the home dir as error.log.

I deployed on a new server today, updated the docs. Lets keep running for a while and see what happens to the metrics (I can already see that the new server is faster).

I've added them all but will keep them inactive until I can spot good thresholds https://grafana.wikimedia.org/dashboard/db/webpagereplay-desktop-alerts

I've enabled them First Visual Change and Speed Index for enwiki on Desktop. Gonna wait until tomorrow to make sure everything is ok and then and then for beta, group0 and group 1 too.

A.k.a when I'm happy I feel that every web site is faaast :)

Yep, you are right.

This is better now, let us close it.

I've looked at the RUM metrics and see the same thing there:

I've looked at the RUM data and I'm pretty sure this is a Chrome regression:

8 of augusts correlates to 1.32.0-wmf.16 - log. The other big increase 5th of September I'm not sure I can spot a specific change in the server admin log? @aaron is back on Monday though.

I've deployed Firefox tests permanently on a c4.xlarge T205246 and it looks good so far.

Fixed this early this week. I've removed one c4.large and updated the docs https://wikitech.wikimedia.org/wiki/Performance/WebPageReplay

The alerts fired but we didn't act on them and then 7 days later it got back to the new "normal":

I removed the ones that was unused. Please let me know if you need help with setting up new dashboards!

This was part of:
https://phabricator.wikimedia.org/phame/post/view/117/performance_testing_in_a_controlled_lab_environment_-_the_metrics/

@Gilles I've added the User Timing the WebPageTest dashboard: https://grafana.wikimedia.org/dashboard/db/webpagetest-drilldown?panelId=47&fullscreen&orgId=1

Or rather just move the enwiki tests to one of the others, that's the easiest way to just test.

It doesn't depend on browser version. Let me try to deploy on another server.

There's bug in the script I created, I'll fix that upstream in Browsertime. For now I added an extra element, the CentralNotice div. I'll test it out on the server running the English wikipedia.

Thank you @Aklapper !

Yep the metrics are more unstable with Chrome 69 on the Chinese wiki by time (haven't calculated by %). For all wikis the mdev is higher with 69 but most wikis (I think all except the English Wikipedia) still have stable medians.

I've updated the settings: Removed the ones that wasn't used, added new ones that are interesting for us.

Yeah, we can try that maybe. I think like this: With the mobile site we can see that we have very low variation (see https://phabricator.wikimedia.org/T185186#4597159). I think some of our pages/wikis are harder to test (more JavaScript before first paint etc) and they are more sensitive to what happens on the server running the tests. But it also depends on the browser, I think I can see that with Chrome 69 on the Chinese wiki we get more unstable metrics (reverting to 68 helped us), I will let that run during the day today and then we can look at the metrics.

Upstream: https://bugs.chromium.org/p/chromium/issues/detail?id=887205

I've made more tests now and made sure the only change is 69. It seems that 69 spends more time in creating the layout:
Layout : 1214.3 vs Layout : 1836.0

Couldn't see any diff in the metrics (that last green line is the change with orange -> white):

Hmm I wonder .. checking the Chinese wiki, the increase in instability in metrics happened exactly when we pushed Chrome 69:

Looking at other wikis it seems pretty clear that 69 introduced something. Only change in the Docker container at that moment was upgrading to 69 (and FF to 62). The blue annotation line in the graph is when we updated to 69: