Ah great that you created the task @Gilles - let me check first thing tomorrow if I can come up with some people.

Been testing it out and I think we need a change in the yargs parser to make configuration easier: https://github.com/yargs/yargs/issues/1363 - I'll have a look tomorrow and see if I can make a PR.

I'll be running through setup the coming days. Yesterday I tested WebPageReplay mobile, today I'm doing desktop replay, and desktop. Then tomorrow I do the emulated mobile. After that I can verify that metrics and keys looks ok. The only thing then left to test is WebPageTest, and I guess the way to do that is to deploy another agent, so it doesn't collide with the current setup.

Just FYI before I went to sleep yesterday I added the main page to my personal test (following the same standard that we will have when we move our testing, so its just a one-liner to add a new URL):

Did a quick check and there where a couple of accessibility issues that I think should be fixed first, making it harder for screenreader etc. But I'm not sure what kind of setup we have today, if we do any automatically tests for that?

With the lazy loading this is solved. What's good is that Chrome 76 enabled lazy loading and seems some tweets that says Apple is also implementing it. Let me look into that. When 76 is widely used + Apple supports it we maybe could switch to use browser internal lazy loading.

I pushed this on the WebPageTest server this morning and it works now:

Instead of moving to the new setup (I need more time to verify that), I've just enabled the jobs in the crontab on the server. When I see that it works, I'll disable the Jenkins job and then concentrate on the long term solution.

Let me check the numbers again and see what it looks like now. For the initial load this have changed since we lazy load on mobile.

We got the alerts in Grafana (and email) that the was big diff in metrics (since we are comparing with 0).

The jobs in Jenkins stopped working:

This is almost done in https://gerrit.wikimedia.org/r/#/c/performance/synthetic-monitoring-tests/+/520413/

We run 30 fps today, I think that is good enough. We can update the API later (we are still waiting on those security issues in the API :()

FYI: Today when we get an alert email, we look into the issue, but we haven't documented the process and how to "think", lets do that in T225987.

Hi @ayounsi I've tracked this in T225403. The reason is that when we updated to Chrome 75, we could see that it's slower for both WebPageTest and Browsertime/WebPageReplay when we run it in emulated mobile mode. So we gonna keep that task open until we have enough users on mobile that uses 75 and see if we can get the same pattern there, then we will create an upstream issue in Chrome (last week 75 wasn't released yet on Android when I tried it out).

In Chrome 77 it will be possible to "try" it out again: https://cs.chromium.org/chromium/src/components/previews/README

If we could upgrade to 6.2.x that would be great. I've been using it for my projects for a while and the lazy loading of panels out of view really makes large dashboards working better (and faster). We have a couple of dashboards in the performance team that are really slow.

Yep got a small change on Browsertime/WebPageReplay too (one tick):

Re-open since we later this year probably can support iOS devices!

One last thing: It's good to turn on screenshots, that makes easier to understand what happens (--chrome.enableTraceScreenshots). Also I tried to add annotations to the timeline console.timeStamp("Test"); but I cannot get them to be visible in the gui (I see them in the trace log though).

Signing over to you Nick to answer the questions.

At least in theory you can tell the webdriver that you wanna have the control back asap the navigation has started (pageLoadStrategy on the webdriver world).

I've had a go today to first try to automate it, so we can find these kind of things in the future when we have a performance device lab. First @nray for the original, that has been solved right?

Oh @nray sorry I missed this, let me have a look first thing tomorrow.

I've pushed Chrome 75 now, let it run for the day and have another look.

No change in WebPageReplay and I could see that it happened in the same time of Chrome 75 for WebPageTest. On Wednesday I'll roll out 75 for WebPageReplay and I can create an upstream task if we get it there too.

Running both for WPT and WPR

This is probably not practical: When we will create alerts for all Wikis we will do that on one dashboard per Wiki. Having everything (or something smart) on one dashboard is probably too complicated.

https://wikitech.wikimedia.org/wiki/Performance/Runbook/WebPageReplay

Done in: https://wikitech.wikimedia.org/wiki/Performance/Synthetic_Measurement_Experiment_2019

The list is done.

This is one of the ones that we blogged about and Chrome acknowledged.

Yes it is fixed ... but I tested it yesterday and the timeline viewer doesn't support the new timings (fixed that yesterday). So I'm thinking as a long term solution we should remove what I did and ask people to use their local version of Chrome.

We blogged about and get some feedback from the Chrome team that we found something.

Yes we do says @Gilles so closing

When we move to the new setup this will be automatically fixed and will be in Git.

This is fixed now

This works better now when we moved to 30 fps for the video.

T190020 can make it easier to get better metrics.

Let us fix that when we have the new setup.

We will automatically have this when we have the mobile device lab.

I think we should do this differently: On alerts we should pre-populate a link with runs before/after a alert, so you as a user just have a link to click on that automatically fills in the right values into the compare page.

Yeah I've modified our instance but that's is no long term solution. Some time ago I added it back, so we more easily can keep our server update.

Moving to the new setup after the summer we will automatically have it.

When we move this to the new setup all configuration will be in JSON and will be much easier than the current setup.

The simplest way to get the temperature is:

adb -s ZY3222N2CZ shell dumpsys battery | grep temperature | grep -Eo '[0-9]{1,3}'

I've changed so default is 30 instead of null as a hack. If it works good I'll make a PR for the API.

I looked into the code and there is a hidden query fps parameter that you can use. There's two ways to do that:

1. Add support to the WPT API (I can do a PR for that)
2. Hack and set it to 30 fps per default (the same way we hacked setting Visual Elements on all runs.

We got feedback in the upstream issue that there's no way to test today but they will add a flag: https://bugs.chromium.org/p/chromium/issues/detail?id=956685#c13

Looks like you only can set the fps for the video per agent (when you start the agent): https://github.com/WPO-Foundation/wptagent/blob/829a9f8111ae8bcf52c99ff574bcec3c924ebd9e/README.md#video-capturedisplay-settings

I think we can close it and the next time someone tries it out and it doesn't work, let us update the docs!

I've updated Browsertime to use Tracium and a setup that catch more log events, that will work perfect for us. It will work the next time we update.

Hi @nray can we do this in two steps? Since we missed it, I think we should try out first to instrument it. We could use https://developer.mozilla.org/en-US/docs/Web/API/User_Timing_API (we already use that for the startup module)? If we can make sure that we measure it, and push it first, then we get some data from real users, before you push the fix. That way we can make sure it is a real issue today, fix it and enjoy the win of the fix and maybe blog about it :)