Page MenuHomePhabricator

Peter (Peter Hedenskog)
Software Engineer, Wikimedia Foundation

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Sunday

  • Clear sailing ahead.

User Details

User Since
Aug 17 2015, 6:48 PM (298 w, 3 d)
Availability
Available
IRC Nick
phedenskog
LDAP User
Unknown
MediaWiki User
PHedenskog (WMF) [ Global Accounts ]

Recent Activity

Sun, May 2

Peter moved T281253: Switch SpeedIndex to LCP for Chrome alerts from Inbox to Backlog: Maintenance on the Performance-Team board.
Sun, May 2, 11:45 AM · Performance-Team, WebPageTest, WebPageReplay

Fri, Apr 30

Krinkle awarded T281558: Investigate possible performance improvements for Main_Page on mobile a Love token.
Fri, Apr 30, 3:52 PM · Performance-Team
Peter created T281558: Investigate possible performance improvements for Main_Page on mobile.
Fri, Apr 30, 9:56 AM · Performance-Team

Tue, Apr 27

Peter created T281253: Switch SpeedIndex to LCP for Chrome alerts.
Tue, Apr 27, 12:53 PM · Performance-Team, WebPageTest, WebPageReplay
Peter closed T281241: Investigate why the mobile servers at the performance device lab do not report any data as Declined.

Since they are moving the phones I'm not gonna do anything more.

Tue, Apr 27, 12:51 PM · Performance-Team, Performance-Device-Lab
Peter added a comment to T281241: Investigate why the mobile servers at the performance device lab do not report any data.
❯ ps -ef | grep 5037
    0  1920     1   0  7:53AM ??         0:01.29 adb -L tcp:5037 fork-server server --reply-fd 5
  501  2317 10076   0  7:55AM ttys000    0:00.00 grep 5037
Tue, Apr 27, 11:56 AM · Performance-Team, Performance-Device-Lab
Peter created T281241: Investigate why the mobile servers at the performance device lab do not report any data.
Tue, Apr 27, 11:52 AM · Performance-Team, Performance-Device-Lab
Peter closed T281023: Test stopped working for wpr-desktop as Resolved.

I changed the three servers for WebPageReplay and the Graphite server.

Tue, Apr 27, 9:47 AM · Performance-Team, WebPageReplay
Peter closed T280715: Investigate Firefox alerts for WebPageReplay as Resolved.

This is first visual change on Obama. The vertical blu line is when Firefox 88 was rolled out:

Tue, Apr 27, 8:56 AM · Performance-Team, WebPageReplay
Peter closed T281102: Update WebPageTest Firefox version to 88 as Resolved.

Last night it was auto updated.

Tue, Apr 27, 8:54 AM · Performance-Team
Peter closed T281102: Update WebPageTest Firefox version to 88, a subtask of T280715: Investigate Firefox alerts for WebPageReplay, as Resolved.
Tue, Apr 27, 8:54 AM · Performance-Team, WebPageReplay
Peter added a comment to T281087: Investigate alerts searching for Obama.

That release increased downloading the base page with 1 seconds in our tests:


Tue, Apr 27, 8:49 AM · Performance-Team
Peter added a comment to T281123: Add CPU benchmark for synthetic tests.

I've added it for WebPageTest and gonna roll it out for the rest later today.

Tue, Apr 27, 7:11 AM · WebPageTest, Performance-Team, WebPageReplay

Mon, Apr 26

Peter moved T281023: Test stopped working for wpr-desktop from Inbox to Doing: Prio Interrupt on the Performance-Team board.
Mon, Apr 26, 6:33 PM · Performance-Team, WebPageReplay
Peter claimed T281023: Test stopped working for wpr-desktop.

Let me fix it on the other servers.

Mon, Apr 26, 6:31 PM · Performance-Team, WebPageReplay
Peter created T281183: Move all synthetic tests alert limits to hard limits.
Mon, Apr 26, 5:50 PM · Performance-Team, WebPageTest, WebPageReplay
Peter moved T281123: Add CPU benchmark for synthetic tests from Inbox to Backlog: Maintenance on the Performance-Team board.
Mon, Apr 26, 5:48 PM · WebPageTest, Performance-Team, WebPageReplay
Peter created T281123: Add CPU benchmark for synthetic tests.
Mon, Apr 26, 11:45 AM · WebPageTest, Performance-Team, WebPageReplay
Peter created T281103: Update how we measure LayoutShift.
Mon, Apr 26, 9:22 AM · Performance-Team, NavigationTiming
Peter moved T281087: Investigate alerts searching for Obama from Inbox to Doing: Prio Interrupt on the Performance-Team board.
Mon, Apr 26, 9:14 AM · Performance-Team
Peter moved T281102: Update WebPageTest Firefox version to 88 from Inbox to Doing: Prio Interrupt on the Performance-Team board.
Mon, Apr 26, 9:14 AM · Performance-Team
Peter created T281102: Update WebPageTest Firefox version to 88.
Mon, Apr 26, 9:12 AM · Performance-Team
Peter created T281087: Investigate alerts searching for Obama.
Mon, Apr 26, 8:08 AM · Performance-Team
Peter reassigned T281023: Test stopped working for wpr-desktop from Peter to dpifke.
Mon, Apr 26, 7:39 AM · Performance-Team, WebPageReplay
Peter updated subscribers of T281023: Test stopped working for wpr-desktop.

Looking at the log I could see:

[2021-04-24 06:45:56] ERROR: Could not upload to S3 RequestTimeTooSkewed: The difference between the request time and th
e current time is too large.
    at Request.extractError (/usr/src/app/node_modules/aws-sdk/lib/services/s3.js:718:35)

And then the local data was not removed.

Mon, Apr 26, 7:37 AM · Performance-Team, WebPageReplay
Peter added a comment to T281023: Test stopped working for wpr-desktop.

This happened again. Let me look for the root cause.

Mon, Apr 26, 7:24 AM · Performance-Team, WebPageReplay

Sat, Apr 24

Peter created T281023: Test stopped working for wpr-desktop.
Sat, Apr 24, 6:20 AM · Performance-Team, WebPageReplay

Fri, Apr 23

Peter added a comment to T280621: Calibrate connectivity for synthetic tests to match FCP and LCP in Chrome User Experience Report.

These are the current metrics:

Fri, Apr 23, 9:06 AM · Performance-Device-Lab, WebPageTest, WebPageReplay, Performance-Team
Peter added a comment to T280621: Calibrate connectivity for synthetic tests to match FCP and LCP in Chrome User Experience Report.

Let me tune the settings on Monday. Lets start with WebPageTest and WebPageReplay.

Fri, Apr 23, 9:03 AM · Performance-Device-Lab, WebPageTest, WebPageReplay, Performance-Team
Peter added a comment to T280621: Calibrate connectivity for synthetic tests to match FCP and LCP in Chrome User Experience Report.

I created a dashboard where we have the metric for all tools: https://grafana.wikimedia.org/d/N-K4xrXGk/synthetic-testing-calibration?orgId=1

Fri, Apr 23, 9:02 AM · Performance-Device-Lab, WebPageTest, WebPageReplay, Performance-Team

Wed, Apr 21

Peter added a comment to T280715: Investigate Firefox alerts for WebPageReplay.

Lets wait until WebPageTest is updated, then I'll go through them. I checked today but no luck yet, I think the update script runs once a day maybe.

Wed, Apr 21, 7:43 AM · Performance-Team, WebPageReplay

Tue, Apr 20

Peter moved T280715: Investigate Firefox alerts for WebPageReplay from Inbox to Doing: Prio Interrupt on the Performance-Team board.
Tue, Apr 20, 6:15 PM · Performance-Team, WebPageReplay
Peter added a comment to T280715: Investigate Firefox alerts for WebPageReplay.

WebPageTest hasn't updated to 88 yet.

Tue, Apr 20, 6:14 PM · Performance-Team, WebPageReplay
Peter created T280715: Investigate Firefox alerts for WebPageReplay.
Tue, Apr 20, 5:55 PM · Performance-Team, WebPageReplay
Peter moved T279615: Increased first visual change when searching on Obama using the search page mobile from Inbox to Doing: Prio Interrupt on the Performance-Team board.
Tue, Apr 20, 5:45 PM · Performance-Team
Peter moved T280621: Calibrate connectivity for synthetic tests to match FCP and LCP in Chrome User Experience Report from Inbox to Doing: Goal-oriented on the Performance-Team board.
Tue, Apr 20, 5:45 PM · Performance-Device-Lab, WebPageTest, WebPageReplay, Performance-Team
Peter closed T280620: Update to Chrome 90 and Firefox 88 as Resolved.
Tue, Apr 20, 5:42 PM · Performance-Team, WebPageReplay
Peter closed T280657: WebPageReplay tests with Chrome 90 do not work as Resolved.

This was my fault from the beginning setting a directory that Chrome didn't have write privileges to inside the container.

Tue, Apr 20, 1:07 PM · WebPageReplay, Performance-Team
Peter created T280657: WebPageReplay tests with Chrome 90 do not work.
Tue, Apr 20, 12:10 PM · WebPageReplay, Performance-Team
Peter created T280621: Calibrate connectivity for synthetic tests to match FCP and LCP in Chrome User Experience Report.
Tue, Apr 20, 6:48 AM · Performance-Device-Lab, WebPageTest, WebPageReplay, Performance-Team
Peter created T280620: Update to Chrome 90 and Firefox 88.
Tue, Apr 20, 6:42 AM · Performance-Team, WebPageReplay

Fri, Apr 9

Legoktm awarded T279804: Visits to Wikimedia properties should not be used for Google ad targeting (FLoC) a Like token.
Fri, Apr 9, 11:35 PM · fundraising-tech-ops, Patch-For-Review, SRE, Traffic, Privacy Engineering, Privacy
Peter created T279804: Visits to Wikimedia properties should not be used for Google ad targeting (FLoC).
Fri, Apr 9, 7:29 PM · fundraising-tech-ops, Patch-For-Review, SRE, Traffic, Privacy Engineering, Privacy
Peter added a comment to T279615: Increased first visual change when searching on Obama using the search page mobile .

This is back to normal. Look at serverResponseTime (responseEnd - requestStart):

Fri, Apr 9, 6:38 AM · Performance-Team
Peter added a comment to T279575: Build Chromedriver for Raspberry Pi.

Yes version 89 is correct, let me re-install my Raspberry and see how it works out!

Fri, Apr 9, 6:31 AM · Performance-Team, Performance-Device-Lab

Thu, Apr 8

Peter closed T264734: Investigate first visual change increase for NL using Firefox as Declined.

Oops, I didn't act fast enough to have a look and now the data is gone. Looking at the current tests it looks like the difference is 0.07 seconds between runs for First Visual Change and that is ok I think. That Ajax URL is still slower than the rest.

Thu, Apr 8, 2:36 PM · WebPageReplay, Performance-Team
Peter closed T264586: Investigate error spikes as Declined.

We don't need to spend time on this, lets do it if we get the same in the new setup.

Thu, Apr 8, 2:32 PM · Performance-Team, Performance-Device-Lab
Peter closed T277835: Investigate really high in First Visual Change on Android Chrome as Resolved.

All these was caused by the purging. I disabled the purging for now.

Thu, Apr 8, 2:31 PM · Patch-For-Review, Performance-Team, Performance-Device-Lab
Peter added a comment to T279615: Increased first visual change when searching on Obama using the search page mobile .

Its seems to correlate to when we show an empty banner: https://meta.wikimedia.org/w/index.php?title=Special:BannerLoader&campaign=impression_test_clear&banner=impression_test2&uselang=en&debug=false

Thu, Apr 8, 1:00 PM · Performance-Team
Peter closed T275346: Investigate high TTFB for WebPageTest desktop tests as Resolved.

This has self healed and doesn't seem to be an issue anymore:

Thu, Apr 8, 10:39 AM · WebPageTest, Performance-Team
Peter changed the status of T278164: Change WebPageTest to use the apache branch from Open to Stalled.

Waiting on input from legal to see what and how we can run it.

Thu, Apr 8, 10:37 AM · WebPageTest, Performance-Team
Peter closed T277480: Increased first visual change for both Chrome and Firefox on desktop as Resolved.

This solved itself on WebPageTest.

Thu, Apr 8, 10:34 AM · Performance-Team, WebPageTest
Peter added a comment to T279584: Evaluate which reverse tethering solution we should use..

Aha! I can try that, do you have exact instructions? Do you need to have the phones rooted for that?

Thu, Apr 8, 7:08 AM · Performance-Team, Performance-Device-Lab
Peter updated the task description for T278305: Grafana dashboards refresh for synthetic testing tools.
Thu, Apr 8, 6:48 AM · Performance-Team, WebPageTest, WebPageReplay
Peter closed T271314: A better retention policy for crux data as Resolved.

I've changed this a couple of days ago to have a policy of keeping the metrics for 2 years. It not so many metrics and we collect them once a day.

Thu, Apr 8, 6:46 AM · Performance-Team
Peter created T279615: Increased first visual change when searching on Obama using the search page mobile .
Thu, Apr 8, 6:34 AM · Performance-Team

Wed, Apr 7

Peter created T279584: Evaluate which reverse tethering solution we should use..
Wed, Apr 7, 7:45 PM · Performance-Team, Performance-Device-Lab
Peter renamed T279583: Continuously run tests using Chrome against WebPageReplay from Lets run tests using Chrome against WebPageReplay to Continuously run tests using Chrome against WebPageReplay.
Wed, Apr 7, 7:40 PM · Performance-Team, Performance-Device-Lab
Peter created T279583: Continuously run tests using Chrome against WebPageReplay.
Wed, Apr 7, 7:39 PM · Performance-Team, Performance-Device-Lab
Peter created T279582: Continuously run tests using Firefox .
Wed, Apr 7, 7:39 PM · Performance-Team, Performance-Device-Lab
Peter created T279581: Continuously run tests using Chrome .
Wed, Apr 7, 7:38 PM · Performance-Team, Performance-Device-Lab
Peter added a subtask for T279572: Evaluate doing Android performance test using a Raspberry Pi: T279579: Build WebPageReplay for Raspberry Pi.
Wed, Apr 7, 7:33 PM · Performance-Team, Performance-Device-Lab
Peter added a parent task for T279579: Build WebPageReplay for Raspberry Pi: T279572: Evaluate doing Android performance test using a Raspberry Pi.
Wed, Apr 7, 7:33 PM · Performance-Team, Performance-Device-Lab
Peter created T279579: Build WebPageReplay for Raspberry Pi.
Wed, Apr 7, 7:33 PM · Performance-Team, Performance-Device-Lab
Peter added a parent task for T279577: Build Geckodriver for Raspberry Pi: T279572: Evaluate doing Android performance test using a Raspberry Pi.
Wed, Apr 7, 7:27 PM · Performance-Team, Performance-Device-Lab
Peter added a subtask for T279572: Evaluate doing Android performance test using a Raspberry Pi: T279577: Build Geckodriver for Raspberry Pi.
Wed, Apr 7, 7:27 PM · Performance-Team, Performance-Device-Lab
Peter created T279577: Build Geckodriver for Raspberry Pi.
Wed, Apr 7, 7:27 PM · Performance-Team, Performance-Device-Lab
Peter added a subtask for T279572: Evaluate doing Android performance test using a Raspberry Pi: T279575: Build Chromedriver for Raspberry Pi.
Wed, Apr 7, 7:22 PM · Performance-Team, Performance-Device-Lab
Peter added a parent task for T279575: Build Chromedriver for Raspberry Pi: T279572: Evaluate doing Android performance test using a Raspberry Pi.
Wed, Apr 7, 7:22 PM · Performance-Team, Performance-Device-Lab
Peter renamed T279575: Build Chromedriver for Raspberry Pi from Build Chromedriver for Raspberry PI to Build Chromedriver for Raspberry Pi.
Wed, Apr 7, 7:21 PM · Performance-Team, Performance-Device-Lab
Peter created T279575: Build Chromedriver for Raspberry Pi.
Wed, Apr 7, 7:21 PM · Performance-Team, Performance-Device-Lab
Peter created T279572: Evaluate doing Android performance test using a Raspberry Pi.
Wed, Apr 7, 7:17 PM · Performance-Team, Performance-Device-Lab

Apr 6 2021

Peter added a comment to T279311: High TTFB standard deviation using WebPageReplay and Android.

Looking at a specific run it looks like this:

Apr 6 2021, 7:41 PM · Performance-Device-Lab

Apr 5 2021

Peter moved T278933: Try out metrics stability running multiple devices using gnirehtet from Inbox to Doing: Prio Interrupt on the Performance-Team board.
Apr 5 2021, 5:56 PM · Performance-Team, Performance-Device-Lab
Peter created T279311: High TTFB standard deviation using WebPageReplay and Android.
Apr 5 2021, 5:45 PM · Performance-Device-Lab

Apr 1 2021

Peter added a comment to T278933: Try out metrics stability running multiple devices using gnirehtet.

Running testing the Obama page increase the standard deviation (three phones at the same time):
TTFB: 1.67s (±63.00ms), firstPaint: 3.22s (±169.00ms
TTFB: 1.64s (±22.00ms), firstPaint: 3.30s (±136.00ms),
TFB: 1.83s (±178.00ms), firstPaint: 3.46s (±311.00ms)

Apr 1 2021, 12:28 PM · Performance-Team, Performance-Device-Lab
Peter added a comment to T278933: Try out metrics stability running multiple devices using gnirehtet.

And then running one phone that is not rooted:
TTFB: 1.63s (±24.00ms), firstPaint: 2.94s (±58.00ms)
TTFB: 1.63s (±28.00ms), firstPaint: 2.98s (±47.00ms)
TTFB: 1.65s (±26.00ms), firstPaint: 2.96s (±51.00ms)

Apr 1 2021, 9:26 AM · Performance-Team, Performance-Device-Lab

Mar 31 2021

Peter added a comment to T278933: Try out metrics stability running multiple devices using gnirehtet.

Running one rooted phone looks like this:
TTFB: 1.62s (19.00ms), firstPaint: 2.96s (51.00ms)
TTFB: 1.66s (54.00ms), firstPaint: 2.95s (69.00ms)
TTFB: 1.64s (19.00ms), firstPaint: 2.95s (63.00ms)

Mar 31 2021, 3:19 PM · Performance-Team, Performance-Device-Lab
Peter added a comment to T278933: Try out metrics stability running multiple devices using gnirehtet.

I've been trying with three devices (all Moto G5, one rooted) and using throttle 4g. It seems to be something with gnirehtet run (run once per device) that do not work/or I don't understand: it seems like the all phones are not tethering. However starting one instance with gnirehtet autorun makes the job.

Mar 31 2021, 12:45 PM · Performance-Team, Performance-Device-Lab
Peter updated the task description for T278923: Resolved emails sometimes as new email threads and sometimes not .
Mar 31 2021, 10:46 AM · Performance-Team (Radar), observability
Peter created T278933: Try out metrics stability running multiple devices using gnirehtet.
Mar 31 2021, 10:39 AM · Performance-Team, Performance-Device-Lab
Peter created T278923: Resolved emails sometimes as new email threads and sometimes not .
Mar 31 2021, 8:52 AM · Performance-Team (Radar), observability

Mar 30 2021

Peter closed T278731: Performance tests on en.m and desktop user journeys stopped as Resolved.
Mar 30 2021, 5:46 PM · Performance-Team
Peter closed T277523: Investigate difference in TTFB on Android Chrome, a subtask of T274227: Try out running tests on Bitbar, as Resolved.
Mar 30 2021, 5:40 PM · Performance-Team, Performance-Device-Lab
Peter closed T277523: Investigate difference in TTFB on Android Chrome as Resolved.
Mar 30 2021, 5:40 PM · Performance-Team, Performance-Device-Lab
Peter added a comment to T277523: Investigate difference in TTFB on Android Chrome.

I've been running this locally to find out the real problem: Running at home, connecting my Mac through a ethernet connection and then reverse tethering running like this:

Mar 30 2021, 5:40 PM · Performance-Team, Performance-Device-Lab
Peter added a comment to T277523: Investigate difference in TTFB on Android Chrome.

I've been trying to reproduce the issues locally by running gnirehtet and throttling as 4g: By doing eleven runs I usually get a standard deviation between 75-120 ms. But I also get those "long runs" where the standard deviation is 700-800 ms and one/two of the runs has TTFB of 3-4 seconds instead of 160 ms. Haven't been able to see anything more yet though.

Mar 30 2021, 12:11 PM · Performance-Team, Performance-Device-Lab
Peter moved T278305: Grafana dashboards refresh for synthetic testing tools from Inbox to Backlog: Future Goals on the Performance-Team board.
Mar 30 2021, 10:02 AM · Performance-Team, WebPageTest, WebPageReplay
Peter moved T278307: Change second view tests to test multiple pages from Backlog: Maintenance to Backlog: Future Goals on the Performance-Team board.
Mar 30 2021, 10:01 AM · Performance-Team
Peter moved T278307: Change second view tests to test multiple pages from Inbox to Backlog: Maintenance on the Performance-Team board.
Mar 30 2021, 10:01 AM · Performance-Team
Peter moved T278731: Performance tests on en.m and desktop user journeys stopped from Inbox to Doing: Prio Interrupt on the Performance-Team board.
Mar 30 2021, 10:01 AM · Performance-Team
Peter closed T277019: Test simpleRT for reverse tethering traffic on Android as Declined.

Actually SimpleRT is abandon so we should not use it.

Mar 30 2021, 9:20 AM · Performance-Device-Lab
Peter updated subscribers of T278731: Performance tests on en.m and desktop user journeys stopped.

We had a lot of logs on that machine. I added a max log size SystemMaxUse=50M in /etc/systemd/journald.conf and restarted. Lets me know @dpifke if there's a better way or something else I should do!

Mar 30 2021, 7:07 AM · Performance-Team
Peter created T278731: Performance tests on en.m and desktop user journeys stopped.
Mar 30 2021, 6:49 AM · Performance-Team

Mar 26 2021

Peter added a comment to T277523: Investigate difference in TTFB on Android Chrome.

I saw this earlier today: https://github.com/WPO-Foundation/wptagent/issues/407 saying that Chrome (maybe) started to do more requests in the backend at the same time of the tests. I don't have my rooted phone with me today so I couldn't check but I have added 30 s sleep before we start our tests for the browser to settle. It reminds me that the Mozilla team actually sleeps for 20 seconds before starting their tests, since Firefox dowload the white/black listings on each browser start.

Mar 26 2021, 10:10 AM · Performance-Team, Performance-Device-Lab
Peter closed T278412: Test running WebPageReplay, a subtask of T274227: Try out running tests on Bitbar, as Resolved.
Mar 26 2021, 9:00 AM · Performance-Team, Performance-Device-Lab
Peter closed T278412: Test running WebPageReplay as Resolved.

This worked out really well:


Mar 26 2021, 9:00 AM · Performance-Team, Performance-Device-Lab
Peter added a subtask for T272979: Onboard Perf Team to new Alerting Toolset : T278514: Wishlist for AlertManager alerts from Grafana.
Mar 26 2021, 8:35 AM · Performance-Team (Radar), User-fgiunchedi, observability
Peter added a parent task for T278514: Wishlist for AlertManager alerts from Grafana: T272979: Onboard Perf Team to new Alerting Toolset .
Mar 26 2021, 8:35 AM · User-fgiunchedi, Performance-Team (Radar), observability
Peter created T278514: Wishlist for AlertManager alerts from Grafana.
Mar 26 2021, 8:34 AM · User-fgiunchedi, Performance-Team (Radar), observability