I continued to test and added the --init to get the rid of the zombie processes. That worked, but after 24 h (or so) Firefox started to fail again. I have reverted back to the old agent and hope to get some attention upstream.

It looks that 74 gives less variation (but probably not as 72):

Ok Firefox worked when I started the container but something happens and now it looks like this again:

Also enabled 74 for enwiki. I think it looks better on variation but maybe some URLs are slower. Need to run it through the weekend and see what it looks like.

No it was not across all wikis. Testing 74 now on German, Spanish, Japanese, Chinese and Russian Wikipedia

No change for Firefox at the same moment, that indicates that there's no change in the Browsertime side, rather a Chrome problem. I'll try out a beta of 74 and see if something changes.

I went through all the tests I ran before on the Dockerized version and then the Firefox tests worked: http://wpt.wmftest.org/result/190403_4N_CA/

I like logs but the logs for WPT is really heard to follow. Here's a snippet from the container:

Created an upstream issue asking for help: https://github.com/WPO-Foundation/wptagent/issues/253

We have some problems on the new instance:

Updated how to create a new container and how to deploy it.

I enabled the alerts on AWS, everything looks good so far except that the used disk space increased by 6% since yesterday, that needs to be fixed.

Let do the blog post tomorrow.

Hmm I think the ScriptDuration is reported in seconds? Yes that much be it.

It is deployed and the tests are running. For Desktop it looks like this (the red line is when I did the switch):

I'm testing https://chromedevtools.github.io/devtools-protocol/tot/Performance to get performance metrics at the moment on my local machine and for the Barack Obama page and the time spent running JS looks like this (headless in Chrome using Fresnel):
ScriptDuration: 0.413074

This is the list so I remember everything when we do the switch:

1. Deploy a new AWS server and update it
2. Install Docker
3. Stop the current agent at AWS (but keep it until we see everything works ok)
4. Remove the current Docker test setup from location.ini
5. Turn off auto updates in settings.ini on the WebPageTest server
6. Start the Docker agent
7. Setup alerts in AWS for the new server
8. Verify that the documentation reflects what I did

I finally got a version up and running on AWS. I've updated the documentation and need to add Docker agent section I will finish that today, and then I think we should make the switch on Monday (then I can also mention it at todays SoS so reading knows before IF there will be any changes to metrics.

This worked fine yesterday so let us close it!

I've moved the dashboards for WebPageTest and WebPageReplay to performance. Signing over you @Krinkle and you can choose how wanna do with the ones for Navtiming.

Added it to both WebPageTest and WebPageReplay, they are under static:
https://grafana.wikimedia.org/d/000000572/webpagereplay-drilldown?orgId=1&var-wiki=static&var-device=desktop&var-browser=chrome&var-latency=100&var-page=Banksy

This is waiting on getting banners on the WebPageReplay tests so I can add the correct alert + verify that it works.

For Chrome we could see different behavior but nothing concistent. I've added annotations on all places so if we see something in RUM we can go back to the synthetic.

This was done last week.

I've tested a couple of URLs locally instead. 11 runs for each URL:

Cool thanks @Krinkle !

Updated.

The current run on WebPageTest got Firefox 66 so I marked an annotation. However we don't graph navigation timing data (or collect it by default) so I think we need to compare individually runs.

@dr0ptp4kt in https://blog.chromium.org/2019/03/chrome-lite-pages-for-faster-leaner.html is says it supports the https://developers.google.com/web/updates/2018/09/reportingapi (adding a URL as a report header) but I haven't looked into it.

Adding example HARs for 66 and 65.

For me it looks like this, the popup happens when I force 2g connections, so not sure when it happens for real users:

I've pushed a fix to fetch the User Timings, let it run for a an hour or two and I can start to add it to the test for group 1 that runs banners at the moment.

I've added it for WebPageTest yesterday and will add it for WebPageReplay the next time we got banners (it will make it easier since we auto-magical pick up the User Timings).

Closing since the privacy issues in T218618 is the most important ones. Also adding https://gist.github.com/soulgalore/430cee73ca2311e815b0fa824b5e0d2a that is a potential (but not working) way to automate to test Lite pages.

Created T218618 about the privacy issue.

Hmm when I tested now from another location it actually works with the navigation/search see:

Btw this reminds me of the problem we used to have with the Opera in proxy mode.

Had a quick at trying to enable the lite pages by command line but it didn't work for me. Tried with

--data-reduction-proxy-lo-fi=always-on --enable-data-reduction-proxy-lite-page --force-effective-connection-type=2G

Yes you are right @Krinkle they are not there. I haven't got around to hook up the phone to my computer yet, let me try that to see if I can see more what's actually is happening.

It looks like this:

However the user can revert by pressing the "Lite"button in the URL bar. If I can find away to share my screenshot from my device ...

Adding @Jdlrobson since I think you will be interested in this.

Had a quick try on 73 beta (73 hasn't reached the play store yet for me). Enable data saver and setting 2g as network let me get the light version of Wikipedia (or one version, I guess it can be multiple versions). Most images are blocked (not all but all major images). Clicking on the them open the media viewer but you cannot see the image. It is also blocked in the Commons File page.

I think the best way to know this is to use the User Timing mwCentralNoticeBanner that will always be there when we show it right @Gilles?

Tried and it was at least so stable so we can measure difference between a logged in user vs a user with a cold cache. I could confirm on a real phone that on a slow connection up:768 down:1600 rtt:150 the first paint happens faster for a logged in user than for a user with a cold cache.

I tried this today from my Mac. The rust version didn't work for me (I got https://github.com/Genymobile/gnirehtet/issues/136). Switching to the Java version worked fine.

Pushed it to https://cloud.docker.com/u/wikimedia/repository/docker/wikimedia/wptagent/general

Let us start with:

Adding how to get Chrome/Firefox versions:

Since there are no changelong and really rare releases, I think the tag needs to include the build date + Chrome and Firefox version (and then we can add Edge (or what it will be called) version when it's available on Linux.

I think the way forward is to tag our own containers. Let me fix that.

We do but only using WebPageReplay: https://grafana.wikimedia.org/d/000000431/webpagereplay?refresh=15m&orgId=1&var-wiki=ruwiki&var-device=desktop&var-browser=chrome&var-latency=100

I like the approach Github has gone with the beta of Github actions. What I like about it, is how super easy it is to integrate whatever tool you need in the pipeline, as long as you have it as a Docker container and you set a couple of labels. I tried it with a couple of projects and it was amazing how easy it was to add new tools. However I understand there are other considerations too :)

So then if I wanna try it out, it should just work if I compare 72 with 73 right? WebPageTest will just roll forward with 73 when it's available but we could run it manually with the current beta installed on the WPT server, just to get the different HAR files, that would be interesting.

To be clear, do you follow/render the redirect or just toss the response once you see the cookie headers? If it's the former, is the browser cache cleared after the main page or is it also not cleared?

I've started documenting at https://wikitech.wikimedia.org/wiki/Performance/Synthetic_Measurement_Experiment_2019 - I'll make sure I'll copy the full scripts by the end of the day and them too and then shutdown the instances and start to add graphs and summary when I'm back at work next Friday.