The private thing fixed it. I've also hardcoded so you can search 31 days back in time.

This done now, but I cannot see any change for Firefox. One thing left todo is to use verify that FF doesn't any extra requests during loading the page, I've only verified that on Mac not Linux.

Two problems here: When I changed to the new setup all tests runs as private by default. I think that hides them from the search. The other is that I've updated the code to run the same as the upstream repo but theres limit there to search only for the last 7 days. That should have been made configurable but it hasn't.

T213833 was the last time this happens, let me go through it again.

See https://github.com/WPO-Foundation/wptagent/issues/284 for the tracking-protection URLs when we compare using Firefox.

There's no way to run from an Android device in EU, but I'm gonna do a manual testing tomorrow.

Compare: https://webpagetest.org/video/compare.php?tests=191202_02_a5854d822511780a53a199c133c20b79,191202_60_532288fb438a800af3460859af628a2e

ATS: https://webpagetest.org/result/191202_02_a5854d822511780a53a199c133c20b79/
nginx: https://webpagetest.org/result/191202_60_532288fb438a800af3460859af628a2e/

Feedback received at the offsite.

@Krinkle I created a new dashboard and missed update Icinga.

The interesting thing is number of processes running on the server from Annie Sullivans performance.now() talk.

I've made some tests so we have some data to compare with. I've accessed https://en.m.wikipedia.org/wiki/Barack_Obama on my Alcatel One (using my office wifi) and then clicking on the navigation to open it, and measure that input delay using the Chrome API First Input Delay. The click always happens after onload.

When getting FID on a server, setting the Chrome CPUThrottlingRate doesn't help. It will not change the FID result emulating a slower device, so we need to do this on a real mobile phone in the future. Let TBT and maxPotentialFid be it for now.

@Gilles I've did some testing. First I tried google.com on my Alacatel One phone. Just accessing the URL and measuring the maxPotentialFID. The thing is Google ship quite much JavaScript, it was different between runs but something like 700-800kb unpacked. For five runs the maxPotentialFid on that phone was: 462, 512, 502, 493, 590 ms.

The missing data problem has been solved since I moved the tests to the new Graphite setup. I've manual updated the dashboard a couple of weeks ago to use the new datasource.

Yes I could that. The way I'm thinking will be the parse time of JavaScript (so long tasks/max potential fid), is that how you think too?. So I'm gonna do a simple page with a lot of HTML + one with only limited JS and run them with Alacatel phone. I'll do that tomorrow.

Re-opening because I still this on some pages: https://s3.amazonaws.com/synthetic-tests-result-wikimedia/ar.m.wikipedia.org/2019-11-25-10-46-36/pages/ar.m.wikipedia.org/wiki/---------/index.html#waterfall

I've updated to a new version that should fix this but lets see what happens with the graph.

Made an upstream issue: https://github.com/WPO-Foundation/webpagetest/issues/1313

I've checked the server and the search logs looks ok. It's setup as it should, I think something changed upstream.

This is fixed in the next update.

I still see this (and get 404 in the test log). https://s3.amazonaws.com/synthetic-tests-result-wikimedia/en.m.wikipedia.org/2019-11-20-17-42-33/pages/en.m.wikipedia.org/wiki/Barack_Obama/index.html#waterfall

The render line was hidden in the overrides

The annotation was missing the $location field fixed now.

Discussed with @Krinkle on our offsite: let us get the metrics in the backend for our user and lets add two tests: One new page in the user space for enwiki where we just add a new date for every save and then in beta we do a save on a large article where change/insert a date in the beginning of the article.

FYI: I've finally got around to measure the CPU time when clicking on the navigation button on mobile. These seems like a perfect opportunity to measure before and after the switch. I haven't added that to our test servers yet so it happens automatically but I can do that. But the best way to measure it is on a real phone. I got a Alcatel One that is perfect for this. When I get back home I'll collect some metrics + look at measure it continuously.

Adding the "new" limits we have them:

When I look at the ones on desktop that has a slow First Contentful Paint, most of the example URLs are on the are from ar.wikipedia.org. One example URL is https://ar.wikipedia.org/wiki/فيسبوك

I've pushed a change so we measure TBT and maxPotentialFid and see what kind of metrics we get from the AWS machines.But I this thing really pushes the need for our performance mobile lab.

I did some tests on the Obama article. Tested on both Alacatel and Moto G5 and it depends, but I've seen times where it 200-288 ms, so really close. And definitely put us on the yellow/orange section. I need more time to test, but its definitely one thing that also do not work in our favor.

Let me try out opening sections as you proposed @Gilles

We can get the trace log and have a look to be 100% sure but I'm pretty convinced its parsing the JavaScript? Since maxPotentialFid takes long tasks after contentful paint and then we parse the JS since we aren't depending on JS to render.

Ok. I've spent some more time on this. The Chrome team also uses something called "Max potential First Input Delay" and that is a CPU long task that happens after first contentful paint, with the idea that the user potentially want to interact with the screen since we show something.

I've spent the day investigating how we can measure FID. It works for me with clicking on a key or mouse click (or finger or mobile). I cannot get it to work with swiping mobile though. I've concentrated measuring FID when everything has downloaded.

If we just care about Chrome we can use the performance observer for
first-input, I've used it here - it supports buffered too so we don't have to add things in head.

If I remember correctly i saw different patterns from the synthetic testing. Some URLs slower, some faster. The good thing when I pushed our new setup we are still running 77, so let me update to 78 to see if can get some more help there.

Did a test using running WPT with native connectivity looking at the stdev:

The old setup is stopped and we only use the new now.

All the alerts are moved too now and I turned off the old setup.

Finished mobile-2g so now its only the actual alerts that needs to be fixed, then we can turn off the old runs.

Got half of https://grafana.wikimedia.org/d/000000205/mobile-2g to be done and the https://grafana.wikimedia.org/d/000000318/webpagetest-alerts

Converted:
https://grafana.wikimedia.org/d/000000057/webpagetest-drilldown
https://grafana.wikimedia.org/d/000000209/wikidata-webpagetest
https://grafana.wikimedia.org/d/000000146/webpagetest-portals
https://grafana.wikimedia.org/d/000000130/mobile-webpagetest

I think is fixed, it was caused because I removed some dashboard and forgot to update Icinga.

And https://wikitech.wikimedia.org/wiki/Performance/Regressions
https://wikitech.wikimedia.org/wiki/Performance/WebPageReplay/Alerts
https://wikitech.wikimedia.org/wiki/Performance/Metrics#WebPageReplay

Done:
https://wikitech.wikimedia.org/wiki/Performance/WebPageReplay
https://wikitech.wikimedia.org/wiki/Performance/Runbook/WebPageReplay

Ah I see now, ok so the alerts we sent have a connection to the dashboard? I forgot about that. Let me update that.