Hi, I've been using M1 arm64 for a while and the workaround --platform linux/amd64 almost never works (at least for me) I think we need to have arm64 builds. I've rebuilt my projects that used Chrome and Firefox to build both amd/arm containers.

Hi @kostajh cool let us talk about this in our weekly meeting on Tuesday and get back to you.

Here's what happens with Largest Contentful Paint. Lets run it one more iteration and then revert the change.

Meeting done.

Sent the email earlier today. Lets keep this open until we get an answer.

I pushed a new version this weekend where the base image is based on Ubuntu 20 and that works.

I haven't been able to fix/understand the root cause. The difference is that the base Docker container uses latest Ubuntu 22 (instead of 20). However it works on my machine and other cloud hosts.

I think what I need to know if if we can have servers running in our dc on an isolated network, maybe you know this @dpifke ? Those servers needs to be able to connect to internet but we do not want them to be connected in any way with the rest of our servers.

Adding some more screenshots. This is what it looked like for the Sweden article for Firefox on WebPageReplay:

Emptied old logs and we are down to 76%, I think that is ok for now.

Thanks a lot @fgiunchedi !!!!

https://grafana.wikimedia.org/d/gVqgE_Q7z/first-view-vs-warm-cache-view

What's the preferred way to send the public key?

Thanks @fgiunchedi and @Dzahn I'll do that first thing next week.

The names in Graphite and the tests are renamed. Gonna let them tun through and verify that they are correct and then lets make the dashboard.

I've took this down to 85% and then moved on with T307093. Let me continue to look when I'm done with the new dashboard.

Actually the trigger job stopped working, When I upgraded to Chrome 101 also the container was updated to latest Ubuntu at it get these errors:

Thank you @colewhite !!!

All was fixed by recreating the alerts.

I'll just check Chrome vs Safari on mobile. When 100 rolled out I saw this https://phabricator.wikimedia.org/T305122#7838322 on WebPageTest, probably because of the hidden Chrome bug (https://bugs.chromium.org/p/chromium/issues/detail?id=1293151 ) about optimise AI.

@colewhite thank you I missed that. Changed so we proxy http to http.

There's an upstream bug (that I do not have access to) https://bugs.chromium.org/p/chromium/issues/detail?id=1293151 and then there's some info in this thread: https://twitter.com/_tbansal/status/1493335460331933696

As reported in T305572 I could see that Chrome makes multiple requests to optimizationguide-pa,googleapis.com. In Browsertime/sitespeed.io those requests are blocked but since we haven't done any update to WebPageTest in a year or more, WebPageTest missed the Chrome flag to turn it off. I can also see that people reported that the requests happen even with the flag enabled so I just added that domain to blocking.

That fixed the problem:

I did push the change now. The Chrome flag I added was --disable-fetching-hints-at-navigation-start and I also blocked that domain to be 100% sure. I'll have a look later tonight and check if it fixed the problem.

I found the root cause by looking at the waterfall graph (the waterfall graph explains how request/responses are handled):

I could also find a jump in responseEnd but only for emulated mobile:

Hi @cjming that seems to correlate when Chrome 100 rolled out on WebPageTest. I found that by using the drill down dashboard for WebPageTest:
https://grafana.wikimedia.org/d/000000057/webpagetest-drilldown?orgId=1&from=1648564576262&to=1648596853128&var-base=sitespeed_io&var-path=webpagetest&var-group=en_wikipedia_org&var-page=_wiki_Barack_Obama&var-browser=&var-location=us-east-chrome&var-connectivity=cable&var-view=firstView and then I zoomed into the time when the change first appeared. Then I click "Show each test" and wait a second. Then the meta data for each test run will load and you will see green vertical line, if you hover on them, you will see a screenshot and browser version (browser version is at the top right). The tests before that test used Chrome 99:

The tests are up and running: https://grafana.wikimedia.org/d/IvAfnmLMk/page-drilldown?orgId=1&var-base=sitespeed_io&var-path=desktopSouthAfrica&var-testtype=firstView&var-group=en_wikipedia_org&var-page=_wiki_Barack_Obama&var-browser=chrome&var-connectivity=native&var-function=median&var-s3path=https:%2F%2Fsynthetic-tests-result-wikimedia.s3.amazonaws.com

I also added mobile version of those two pages. Let me verify that the tests continues to run through the day and then this is done.

I deployed a server this morning, we send the result to Graphite and AWS. At the moment we tests https://zu.wikipedia.org/wiki/Ikhasi_Elikhulu https://en.wikipedia.org/wiki/Barack_Obama on native (data centre speed) and cable + I added the login journey so we can see the difference against running in US. I'll also add some emulated desktop tests for completeness.

Needs T278172 to be able to move on.

That was done for Kobiton.

And I've pinged them again.

I think this was something wrong with Kobitons setup, this works just fine on my own setup and for Mozilla.

That was done when things where running on Kobiton.

No need to fix this until we have something up and running again.

Lets do this if we are feel running with our root get us to unstable metrics.

I created https://www.sitespeed.io/documentation/sitespeed.io/mobile-phones/#on-your-phone a couple of years ago. When we push a new device lab, we can add our own docs following that example.

Also I got the alarm email but I missed it.

The problem is the worker that kicks off the jobs:

We will not do this now since the change of license.

I hacked this again setting the limit to 100 days. I don't think we should do an upstream fix,

This has been fixed but nothing done at our side.

Lets do this if its ever needed again, no need to spend time on that now.

Since the change of the license of WebPageTest to a non open source license, this is nothing we want to do.

Long time overdue, nothing to do here now.