Page MenuHomePhabricator
Create Task
Maniphest T133646

Run performance test on commits (Fresnel)
Closed, ResolvedPublic
Actions

Description

In the current setup with WebPageTest we continuously test a couple of pages on different hosts. But we don't test commits. The long term goal should be to continuously and automatically do performance tests on new commits.

A first step could be to do one of the things that Facebook do: Make it possible for a developer to easily test her/his changes and get back median metrics with a confident value.

To make that happen there's a lot we need to do:

  • We should have the test infrastructure in house and not on Amazon to make sure we have full control, to be able to minimize variance.
  • We should prepare for testing on real mobile devices.
  • Dedicated hardware to run the tests (we should have both desktop and mobile testing)
  • Run the tests on commits
  • GUI driven, whatever tools we use, it must be super easy to start a test and easy to understand the result = should the code move on or not.

See also:

Details

ProjectBranchLines +/-Subject
mediawiki/coremaster+17 -0build: Add initial version of Fresnel config
integration/configmaster+26 -1zuul: Enable fresnel job in a new 'patch-performance' pipeline
integration/configmaster+1 -1jjb: Update Fresnel job to quibble-fresnel:0.0.28-5
integration/configmaster+10 -0Set MW_SERVER/MW_SCRIPT_PATH for fresnel
integration/configmaster+1 -1jjb: Update Fresnel job to quibble-fresnel:0.0.28-4
integration/configmaster+14 -1add firefox-esr to fresnel, to make chromium work (!)
performance/fresnelmaster+0 -20build: Remove the MediaWiki .fresnel.yml file
integration/configmaster+1 -1jjb: Update Fresnel job to quibble-fresnel:0.0.28-3
integration/configmaster+10 -0Set CHROMIUM_FLAGS in fresnel image
integration/configmaster+1 -1fresnel: Fix syntax error
integration/configmaster+1 -1jjb: Update Fresnel job to quibble-fresnel:0.0.28-2
integration/configmaster+12 -5Add missing node-gyp deps to fresnel image
integration/configmaster+4 -0Add fresnel to 'experimental' pipeline for mediawiki/core:master
integration/configmaster+491 -0Add mediawiki-fresnel-patch job
performance/fresnelmaster+203 -297Require Node 10 and adopt async-await
performance/fresnelmaster+8 K -0initial commit
integration/configmaster+19 -1zuul: Add custom DOC_PROJECT destination for fresnel
integration/configmaster+23 -1Create Node 10 coverage-publish job and use for Fresnel
integration/configmaster+25 -4Switch Fresnsel from node10-test to node10-browser-test
integration/configmaster+24 -3Create node-10-docker job template and use for Fresnel
integration/configmaster+9 -2Switch fresnel Jenkins job from 'npm' to 'npm-with-browser' image
integration/configmaster+16 -12Add pipeline for performance/fresnel.git (npm-docker)
mediawiki/coremaster+177 -9resourceloader: Implement mw.inspect 'time' report
Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Krinkle moved this task from Backlog: Maintenance, non-prioritized to To-do: Goals, prioritized next 4 Quarters on the Performance-Team board.Jun 14 2017, 7:25 PM
• Gilles closed subtask T153360: [Research] Proxy for local performance testing as Resolved.Jun 21 2017, 1:24 PM
Peter closed subtask T153421: Set connectivity for testing on commits as Resolved.Sep 12 2017, 3:07 PM
Krinkle mentioned this in T153044: Simplify local performance testing on real device by setting connectivity.Sep 12 2017, 4:40 PM
Peter created subtask T176361: Run performance tests using local proxy.Sep 20 2017, 8:58 PM
• Gilles closed subtask T176361: Run performance tests using local proxy as Resolved.Dec 20 2017, 9:19 AM
• Imarlier moved this task from To-do: Goals, prioritized next 4 Quarters to Backlog: Maintenance, non-prioritized on the Performance-Team board.Jan 18 2018, 3:42 PM
Krinkle updated the task description. (Show Details)Jun 20 2018, 1:56 PM
Krinkle mentioned this in T136850: Test performance on real mobile devices.
Krinkle moved this task from Backlog: Maintenance, non-prioritized to To-do: Goals prioritized current Quarter on the Performance-Team board.Jun 21 2018, 9:39 AM
• Imarlier closed subtask T153069: Detect performance regressions when comparing timelines/profiles as Declined.Jun 21 2018, 10:00 AM
Krinkle mentioned this in T191617: Mobile performance testing alternatives.Jun 21 2018, 11:27 AM
Krinkle claimed this task.Aug 22 2018, 2:19 AM
Krinkle moved this task from To-do: Goals prioritized current Quarter to Doing (old) on the Performance-Team board.
Krinkle added a subscriber: Krinkle.
gerritbot added a subscriber: gerritbot.Aug 22 2018, 2:20 AM

Change 444773 had a related patch set uploaded (by Krinkle; owner: Krinkle):
[mediawiki/core@master] resourceloader: Implement mw.inspect 'time' report

https://gerrit.wikimedia.org/r/444773

gerritbot added a project: Patch-For-Review.Aug 22 2018, 2:20 AM
gerritbot added a comment.Aug 22 2018, 4:24 AM

Change 454465 had a related patch set uploaded (by Krinkle; owner: Krinkle):
[mediawiki/core@master] [WIP] resourceloader: Add rIC and $.ready tracking to mw.inspect

https://gerrit.wikimedia.org/r/454465

Krinkle added a subscriber: ori.EditedAug 22 2018, 7:37 PM
@ori wrote at https://gerrit.wikimedia.org/r/#/c/mediawiki/core/+/444773/

Very cool. I am wondering why you chose to make this off-by-default, though, since one use-case is "why did this article I just loaded take so long to load?"., and profiler.js is pretty tiny.

(Responding here for visibility as others might wonder the same, and because I got a bit side-tracked into other stuff to talk about.)

A couple reasons for off by default:

  1. I've already encountered and had to fix dozens of subtle bugs in just this simple initial version, so until more experience, I'm not letting it go in prod breaking things in hard to detect ways. Not initially anyway.
  1. I'm not too worried about the byte size actually, and more about run-time overhead and memory use. On that topic, something I've come to dislike is memory size of the registry object, in particular from the styes and scripts retention (copy of implement payload). These are unused after execution, but were not removed. They were removable, until introduction of mw.inspect.grep, which is useful, but makes the registry quite big. The same also applies to the full-size copy of unserialised mw.loader.store. I agree these are useful for debugging in production, and it is useful to be able to debug an already loaded page. But the underlying issue is in my opinion that our actual "debug" mode is not useful. I haven't used it years, and actively recommend others to avoid it. Its behaviour doesn't match prod, many features are unavailable, it's slow, it's inefficient etc. It's one of my regrets to not have addressed earlier. Part of T85805 (maybe next quarter/next year?), I'd like to re-implement debug mode, starting with plain prod mode, unchanged, except for minification (perhaps), and a few additive features (no behaviour change). By being useful and low in overhead, it also addresses the issue of being able to instrument an already loaded page, as one could just leave it on by default for yourself (via cookie). After that, we could consider enabling profiling by default (in debug mode). And optimise prod mode to more actively deference parts of the registry.
  1. This new profiling feature is, right now, mainly for perf-testing on commit. The rough idea is to add a Jenkins job that will install MW twice (before/after) and extract various metrics and present the median, stddev, and difference between runs of those metrics, and fail the build on exceeding of certain thresholds. Somewhat similar to what Lego recently introduced for code coverage. These module-level metrics would be part of the collected dataset.
  1. This feature's secondary purpose is to help with the initial audit of various features' overhead on page load time. This is mostly low-hanging fruit that can be easily demonstrated and worked on from one-off profiling locally. I've been doing this manually for a few months, but having a published method will help make it more visible to others. To collectively cut down time-to-interactive. Much of it is/was in WikimediaEvents – expensive DOM queries, large node lists, attaching 100s/1000s of event listeners, unconditionally pre-computing cryptos – all blocking the module from reaching state "ready" and executing the next module. Addressing these issues typically involves making things lazy/conditional, adopting delegate events where possible, or moving to dom-ready, window-load or requestIdleCallback, etc.
gerritbot added a comment.Aug 23 2018, 6:19 PM

Change 444773 merged by jenkins-bot:
[mediawiki/core@master] resourceloader: Implement mw.inspect 'time' report

https://gerrit.wikimedia.org/r/444773

ReleaseTaggerBot added a project: MW-1.32-notes (WMF-deploy-2018-08-28 (1.32.0-wmf.19)).Aug 23 2018, 7:01 PM
Peter added a comment.Aug 27 2018, 7:48 PM

We should have the test infrastructure in house and not on Amazon to make sure we have full control, to be able to minimize variance.

The ironic thing here is that when Gilles and me tried to get as stable metrics as possible with different proxies Amazon gave us far better numbers and minimized variance much more than our own infrastructure. I think we need help with this before we move on.

Krinkle added a comment.Aug 28 2018, 12:45 AM

@Peter Yeah, I expect that values between runs will be unstable. However, I'm hoping it will at least be stable back-to-back when quickly running "before" and "after" in the same Jenkins job – which is my current thinking for commit testing.

We can't store or plot the metrics between builds in a useful way anyway because there will not be a logical progression between builds (each build is for a different unrelated repository, branch and commit that is still being reviewed, not merged).

But, it is possible that even within the same build doing two runs after each other will be unstable. I'll find out soon :)

Peter added a comment.Aug 29 2018, 11:48 AM

I see your point. It will be really interesting to see the result!

gerritbot added a comment.Sep 9 2018, 3:55 AM

Change 459268 had a related patch set uploaded (by Krinkle; owner: Krinkle):
[integration/config@master] [WIP] Add performance-patch job

https://gerrit.wikimedia.org/r/459268

gerritbot added a comment.Sep 9 2018, 11:48 PM

Change 459409 had a related patch set uploaded (by Krinkle; owner: Krinkle):
[performance/fresnel@master] initial commit

https://gerrit.wikimedia.org/r/459409

Nikerabbit added a subscriber: Nikerabbit.Sep 10 2018, 7:21 AM
Krinkle mentioned this in T187684: Define key metrics (3-5 total) that we're tracking and go public with the metrics.Sep 28 2018, 2:20 AM
gerritbot added a comment.Oct 2 2018, 3:24 AM

Change 463886 had a related patch set uploaded (by Krinkle; owner: Krinkle):
[integration/config@master] Add pipeline for performance/fresnel.git (npm-docker)

https://gerrit.wikimedia.org/r/463886

gerritbot added a comment.Oct 2 2018, 3:25 AM

Change 463886 merged by jenkins-bot:
[integration/config@master] Add pipeline for performance/fresnel.git (npm-docker)

https://gerrit.wikimedia.org/r/463886

Peter added a comment.Oct 17 2018, 6:03 AM

Did you see https://tech.trivago.com/2018/10/12/building-fast-and-reliable-web-applications/ @Krinkle

They use Puppeteer to collect some info, Lighthouse for some and run sitespeed.io for some (but they do not graph the one from sitespeed.io what's up with that?). All running in a Jenkins setup.

Krinkle moved this task from Doing (old) to To-do: Goals prioritized current Quarter on the Performance-Team board.Nov 5 2018, 10:06 PM
gerritbot added a comment.Dec 17 2018, 10:37 PM

Change 480256 had a related patch set uploaded (by Krinkle; owner: Krinkle):
[integration/config@master] Switch fresnel Jenkins job from 'npm' to 'npm-with-browser' image

https://gerrit.wikimedia.org/r/480256

gerritbot added a comment.Dec 17 2018, 10:45 PM

Change 480256 merged by jenkins-bot:
[integration/config@master] Switch fresnel Jenkins job from 'npm' to 'npm-with-browser' image

https://gerrit.wikimedia.org/r/480256

Krinkle moved this task from To-do: Goals prioritized current Quarter to Doing (old) on the Performance-Team board.Dec 19 2018, 10:51 PM
gerritbot added a comment.Jan 7 2019, 12:42 AM

Change 482572 had a related patch set uploaded (by Krinkle; owner: Krinkle):
[integration/config@master] Create node-10-docker job template and use for Fresnel

https://gerrit.wikimedia.org/r/482572

gerritbot added a comment.Jan 7 2019, 7:55 PM

Change 482572 merged by jenkins-bot:
[integration/config@master] Create node-10-docker job template and use for Fresnel

https://gerrit.wikimedia.org/r/482572

gerritbot added a comment.Jan 7 2019, 8:12 PM

Change 482695 had a related patch set uploaded (by Krinkle; owner: Krinkle):
[integration/config@master] Switch Fresnsel from node10-test to node10-browser-test

https://gerrit.wikimedia.org/r/482695

gerritbot added a comment.Jan 7 2019, 8:28 PM

Change 482695 merged by jenkins-bot:
[integration/config@master] Switch Fresnsel from node10-test to node10-browser-test

https://gerrit.wikimedia.org/r/482695

gerritbot added a comment.Jan 8 2019, 2:03 AM

Change 482749 had a related patch set uploaded (by Krinkle; owner: Krinkle):
[performance/fresnel@master] Require Node 10 and adopt async-await

https://gerrit.wikimedia.org/r/482749

Krinkle renamed this task from Run performance test on commits to Run performance test on commits (Fresnel).Jan 8 2019, 2:33 AM
Krinkle removed a project: MW-1.32-notes (WMF-deploy-2018-08-28 (1.32.0-wmf.19)).
gerritbot added a comment.Jan 8 2019, 2:34 AM

Change 482751 had a related patch set uploaded (by Krinkle; owner: Krinkle):
[integration/config@master] Add Fresnel postmerge job for publishing coverage report

https://gerrit.wikimedia.org/r/482751

gerritbot added a comment.Jan 8 2019, 2:39 AM

Change 482751 merged by jenkins-bot:
[integration/config@master] Create Node 10 coverage-publish job and use for Fresnel

https://gerrit.wikimedia.org/r/482751

gerritbot added a comment.Jan 8 2019, 2:57 AM

Change 482752 had a related patch set uploaded (by Krinkle; owner: Krinkle):
[integration/config@master] zuul: Add custom DOC_PROJECT destination for fresnel

https://gerrit.wikimedia.org/r/482752

gerritbot added a comment.Jan 8 2019, 3:57 AM

Change 482752 merged by jenkins-bot:
[integration/config@master] zuul: Add custom DOC_PROJECT destination for fresnel

https://gerrit.wikimedia.org/r/482752

gerritbot added a comment.Jan 9 2019, 7:33 AM

Change 459409 merged by jenkins-bot:
[performance/fresnel@master] initial commit

https://gerrit.wikimedia.org/r/459409

gerritbot added a comment.Jan 15 2019, 6:59 PM

Change 482749 merged by jenkins-bot:
[performance/fresnel@master] Require Node 10 and adopt async-await

https://gerrit.wikimedia.org/r/482749

gerritbot added a comment.Feb 5 2019, 8:57 PM

Change 488121 had a related patch set uploaded (by Krinkle; owner: Krinkle):
[mediawiki/core@master] build: Add fresnel scenarios config

https://gerrit.wikimedia.org/r/488121

gerritbot added a comment.Feb 5 2019, 9:46 PM

Change 488187 had a related patch set uploaded (by Krinkle; owner: Krinkle):
[performance/fresnel@master] build: Remove the MediaWiki .fresnel.yml file

https://gerrit.wikimedia.org/r/488187

gerritbot added a comment.Feb 6 2019, 8:36 AM

Change 488187 merged by jenkins-bot:
[performance/fresnel@master] build: Remove the MediaWiki .fresnel.yml file

https://gerrit.wikimedia.org/r/488187

Krinkle added a comment.Feb 10 2019, 12:51 AM
In T133646#4569610, @gerritbot wrote:

Change 459268 had a related patch set uploaded (by Krinkle; owner: Krinkle):
[integration/config@master] [WIP] Add performance-patch job

https://gerrit.wikimedia.org/r/459268

This draft revision to the quibble-strech-php70 docker image is building without errors for me locally now, and installs Fresnel from npm.

The problem though, is that the Quibble image ships with Node 6 instead of Node 10 still. So we'll need to fix that first.

Stashbot added a subscriber: Stashbot.Feb 14 2019, 8:50 PM
This comment was removed by Krinkle.
Jdforrester-WMF added a subscriber: Jdforrester-WMF.Feb 14 2019, 8:56 PM
gerritbot added a comment.Feb 18 2019, 9:33 PM

Change 459268 merged by jenkins-bot:
[integration/config@master] Add mediawiki-fresnel-patch job

https://gerrit.wikimedia.org/r/459268

Stashbot added a comment.Feb 18 2019, 9:37 PM

Mentioned in SAL (#wikimedia-releng) [2019-02-18T21:37:18Z] <Krinkle> Updating docker-pkg files on contint1001 for https://gerrit.wikimedia.org/r/459268 / T133646

gerritbot added a comment.Feb 18 2019, 9:48 PM

Change 491378 had a related patch set uploaded (by Krinkle; owner: Krinkle):
[integration/config@master] Add fresnel to 'experimental' pipeline for mediawiki/core:master

https://gerrit.wikimedia.org/r/491378

gerritbot added a comment.Feb 18 2019, 9:50 PM

Change 491378 merged by jenkins-bot:
[integration/config@master] Add fresnel to 'experimental' pipeline for mediawiki/core:master

https://gerrit.wikimedia.org/r/491378

Stashbot added a comment.Feb 18 2019, 9:51 PM

Mentioned in SAL (#wikimedia-releng) [2019-02-18T21:51:09Z] <Krinkle> Reloading Zuul to deploy https://gerrit.wikimedia.org/r/491378 / T133646)

gerritbot added a comment.Feb 18 2019, 10:20 PM

Change 491381 had a related patch set uploaded (by Krinkle; owner: Krinkle):
[integration/config@master] Add missing node-gyp deps to fresnel image

https://gerrit.wikimedia.org/r/491381

gerritbot added a comment.Feb 18 2019, 10:22 PM

Change 491382 had a related patch set uploaded (by Krinkle; owner: Krinkle):
[integration/config@master] jjb: Update Fresnel job to quibble-fresnel:0.0.28-2

https://gerrit.wikimedia.org/r/491382

gerritbot added a comment.Feb 19 2019, 12:53 AM

Change 491381 merged by jenkins-bot:
[integration/config@master] Add missing node-gyp deps to fresnel image

https://gerrit.wikimedia.org/r/491381

Stashbot added a comment.Feb 19 2019, 12:54 AM

Mentioned in SAL (#wikimedia-releng) [2019-02-19T00:54:25Z] <Krinkle> Updating docker-pkg files on contint1001 for https://gerrit.wikimedia.org/r/491381 / bc8e4198961cb73 / T133646

gerritbot added a comment.Feb 19 2019, 12:59 AM

Change 491382 merged by jenkins-bot:
[integration/config@master] jjb: Update Fresnel job to quibble-fresnel:0.0.28-2

https://gerrit.wikimedia.org/r/491382

gerritbot added a comment.Feb 19 2019, 1:02 AM

Change 491394 had a related patch set uploaded (by Krinkle; owner: Krinkle):
[integration/config@master] fresnel: Fix syntax error

https://gerrit.wikimedia.org/r/491394

gerritbot added a comment.Feb 19 2019, 1:05 AM

Change 491394 merged by jenkins-bot:
[integration/config@master] fresnel: Fix syntax error

https://gerrit.wikimedia.org/r/491394

gerritbot added a comment.Feb 19 2019, 1:26 AM

Change 491396 had a related patch set uploaded (by Krinkle; owner: Krinkle):
[integration/config@master] Set CHROMIUM_FLAGS in fresnel image

https://gerrit.wikimedia.org/r/491396

gerritbot added a comment.Feb 19 2019, 3:50 AM

Change 491403 had a related patch set uploaded (by Krinkle; owner: Krinkle):
[integration/config@master] jjb: Update Fresnel job to quibble-fresnel:0.0.28-3

https://gerrit.wikimedia.org/r/491403

gerritbot added a comment.Feb 19 2019, 3:54 AM

Change 491396 merged by Krinkle:
[integration/config@master] Set CHROMIUM_FLAGS in fresnel image

https://gerrit.wikimedia.org/r/491396

gerritbot added a comment.Feb 19 2019, 3:54 AM

Change 491403 merged by Krinkle:
[integration/config@master] jjb: Update Fresnel job to quibble-fresnel:0.0.28-3

https://gerrit.wikimedia.org/r/491403

gerritbot added a comment.Feb 19 2019, 4:47 AM

Change 491405 had a related patch set uploaded (by Krinkle; owner: Krinkle):
[integration/config@master] add firefox-esr to fresnel, to make chromium work (!)

https://gerrit.wikimedia.org/r/491405

gerritbot added a comment.Feb 19 2019, 4:47 AM

Change 491406 had a related patch set uploaded (by Krinkle; owner: Krinkle):
[integration/config@master] jjb: Update Fresnel job to quibble-fresnel:0.0.28-4

https://gerrit.wikimedia.org/r/491406

gerritbot added a comment.Feb 19 2019, 4:49 AM

Change 491405 merged by jenkins-bot:
[integration/config@master] add firefox-esr to fresnel, to make chromium work (!)

https://gerrit.wikimedia.org/r/491405

gerritbot added a comment.Feb 19 2019, 4:52 AM

Change 491406 merged by jenkins-bot:
[integration/config@master] jjb: Update Fresnel job to quibble-fresnel:0.0.28-4

https://gerrit.wikimedia.org/r/491406

gerritbot added a comment.Feb 19 2019, 3:20 PM

Change 491513 had a related patch set uploaded (by Krinkle; owner: Krinkle):
[integration/config@master] Set MW_SERVER/MW_SCRIPT_PATH for fresnel

https://gerrit.wikimedia.org/r/491513

gerritbot added a comment.Feb 19 2019, 3:34 PM

Change 491516 had a related patch set uploaded (by Krinkle; owner: Krinkle):
[integration/config@master] jjb: Update Fresnel job to quibble-fresnel:0.0.28-5

https://gerrit.wikimedia.org/r/491516

gerritbot added a comment.Feb 19 2019, 3:54 PM

Change 491513 merged by jenkins-bot:
[integration/config@master] Set MW_SERVER/MW_SCRIPT_PATH for fresnel

https://gerrit.wikimedia.org/r/491513

gerritbot added a comment.Feb 19 2019, 3:58 PM

Change 491516 merged by jenkins-bot:
[integration/config@master] jjb: Update Fresnel job to quibble-fresnel:0.0.28-5

https://gerrit.wikimedia.org/r/491516

gerritbot added a comment.Feb 19 2019, 5:32 PM

Change 488121 merged by jenkins-bot:
[mediawiki/core@master] build: Add initial version of Fresnel config

https://gerrit.wikimedia.org/r/488121

ReleaseTaggerBot added a project: MW-1.33-notes (1.33.0-wmf.18; 2019-02-19).Feb 19 2019, 6:00 PM
gerritbot added a comment.Feb 19 2019, 8:39 PM

Change 491577 had a related patch set uploaded (by Krinkle; owner: Krinkle):
[integration/config@master] zuul: Enable fresnel job on coverage pipeline

https://gerrit.wikimedia.org/r/491577

gerritbot added a comment.Feb 19 2019, 9:21 PM

Change 491577 merged by jenkins-bot:
[integration/config@master] zuul: Enable fresnel job in a new 'patch-performance' pipeline

https://gerrit.wikimedia.org/r/491577

Krinkle mentioned this in Fresnel.Feb 19 2019, 10:47 PM
Krinkle edited projects, added Fresnel; removed MW-1.33-notes (1.33.0-wmf.18; 2019-02-19), Patch-For-Review.
Krinkle closed this task as Resolved.Feb 19 2019, 10:49 PM

Future development will be tracked under under Fresnel

Krinkle edited projects, added Performance-Team-publish; removed Epic.Feb 19 2019, 10:49 PM
Tgr awarded a token.Mar 10 2019, 6:00 AM
Krinkle moved this task from Untriaged to Ready for write-up on the Performance-Team-publish board.Mar 14 2019, 12:55 AM
Krinkle mentioned this in T221179: Measure time spent in rIC and $.ready with mw.loader.profiler..Apr 17 2019, 12:36 AM
pmiazga mentioned this in T222163: [EPIC] Monitor complex pages rendering time on mobile web.Apr 30 2019, 11:30 AM
Content licensed under Creative Commons Attribution-ShareAlike 4.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL