Page MenuHomePhabricator
Create Task
Maniphest T225416

Unify synthetic testing scripts
Closed, ResolvedPublic
Actions

Description

As I showed in the last off site, we should move and unify all configurations for WebPageTest and Browsertime. Today the config for Browsertime only lives on the servers (huuu) and WebPageTest in Git. Lets add the tests in performance/synthetic-monitoring-tests

I'll create a new structure per server the coming weeks.

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
Remove low prio run for WebPageTest.performance/synthetic-monitoring-testsmaster+3 -6
Bump to 10.3.0.performance/synthetic-monitoring-testsmaster+1 -1
Collect and store CSS/JS/HTML for Firefox tests.performance/synthetic-monitoring-testsmaster+1 -1
Disable videos.performance/synthetic-monitoring-testsmaster+6 -6
Remove Docker volumes for each iteration.performance/synthetic-monitoring-testsmaster+1 -1
Bump sitespeed.io with a fix where the timer video do not flicker.performance/synthetic-monitoring-testsmaster+1 -1
Update to stable release of sitespeed.io.performance/synthetic-monitoring-testsmaster+1 -1
Run job on lower prio than the current test server.performance/synthetic-monitoring-testsmaster+9 -3
Setup a readme, explain what we do.performance/synthetic-monitoring-testsmaster+55 -5
Use correct script name for finding configuration.performance/synthetic-monitoring-testsmaster+4 -4
Config paths missed config dir (depends on where you start the script)performance/synthetic-monitoring-testsmaster+4 -4
Correct variabe name for tests.performance/synthetic-monitoring-testsmaster+2 -4
Run multiple tests after each other on the same machine.performance/synthetic-monitoring-testsmaster+22 -15
Decrease number of runs for Alexa tests to 1.performance/synthetic-monitoring-testsmaster+7 -1
Prepare to run multiple tests after each other.performance/synthetic-monitoring-testsmaster+13 -13
Make it possible to override configurations per test.performance/synthetic-monitoring-testsmaster+25 -6
Split WebPageTest configuration in one per browser.performance/synthetic-monitoring-testsmaster+15 -4
Use the default WebPageTest agent.performance/synthetic-monitoring-testsmaster+2 -2
Move Browsertime Chrome setup to Browsertime section.performance/synthetic-monitoring-testsmaster+7 -7
Remove role::ci::slave::webperformanceoperations/puppetproduction+0 -15
Mobile configuration was at the wrong place.performance/synthetic-monitoring-testsmaster+1 -2
Add missing mobile configuration that sets correct view port & user agent.performance/synthetic-monitoring-testsmaster+1 -0
Bump sitespeed.io version to fix connectivity alias bug.performance/synthetic-monitoring-testsmaster+1 -1
Verify that the input directory for tests really exists.performance/synthetic-monitoring-testsmaster+8 -0
Move tests to multiple files to follow WebPageReplay standard.performance/synthetic-monitoring-testsmaster+14 -16
Use latest Docker container with fixes for annotations using WebPageTest.performance/synthetic-monitoring-testsmaster+1 -1
Remove disable Browsertime.performance/synthetic-monitoring-testsmaster+1 -1
Add Firefox tests.performance/synthetic-monitoring-testsmaster+19 -9
Post test for WebPageTest under the webpagetest namespace.performance/synthetic-monitoring-testsmaster+0 -0
Add meta data to each script to make it to see what we are testing.performance/synthetic-monitoring-testsmaster+13 -0
Also run emulated runs on WebPageTestperformance/synthetic-monitoring-testsmaster+7 -2
Use new test location for WebPageTest agent when we are testing the setup.performance/synthetic-monitoring-testsmaster+2 -2
Enable showing the scipt that runs the tests.performance/synthetic-monitoring-testsmaster+4 -2
Generic hide all elements to avoid submit button to affect First Visual Changeperformance/synthetic-monitoring-testsmaster+6 -0
Renamed structure to not include dot since it broke the namespace handling.performance/synthetic-monitoring-testsmaster+0 -0
Add white/clean the HTML so that FirstVisualChange is ok for mobile logged in.performance/synthetic-monitoring-testsmaster+6 -0
Set alias of connectivity so we can see what actual alias is used.performance/synthetic-monitoring-testsmaster+4 -2
Median wait/read time is 21 seconds so let us run with that.performance/synthetic-monitoring-testsmaster+2 -2
Remove background between page loads to fix early First Visual Change on videos.performance/synthetic-monitoring-testsmaster+6 -0
Remove URLs that block AWS traffic.performance/synthetic-monitoring-testsmaster+0 -2
layout: [performance/WebPageTest] Archiveintegration/configmaster+1 -1
jjb: Drop cron triggered WebPageTest runintegration/configmaster+1 -132
layout: Add tests for synthetic monitoring setup.integration/configmaster+4 -0
Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
gerritbot added a comment.Sep 2 2019, 9:11 AM

Change 533868 merged by jenkins-bot:
[performance/synthetic-monitoring-tests@master] Mobile configuration was at the wrong place.

https://gerrit.wikimedia.org/r/533868

• Peter closed subtask T231413: Fixes for running WebPageTest as Resolved.Sep 3 2019, 6:45 AM
gerritbot added a comment.Sep 3 2019, 10:33 AM

Change 534131 had a related patch set uploaded (by Phedenskog; owner: Phedenskog):
[performance/synthetic-monitoring-tests@master] Setup a readme, explain what we do.

https://gerrit.wikimedia.org/r/534131

gerritbot added a comment.Sep 4 2019, 3:50 AM

Change 531420 merged by Dzahn:
[operations/puppet@production] Remove role::ci::slave::webperformance

https://gerrit.wikimedia.org/r/531420

hashar added a comment.Sep 4 2019, 6:41 AM

All the logic to host the webperformance jobs on the CI infrastructure have been disposed of. I guess that concludes my part of the work :]

gerritbot added a comment.Sep 6 2019, 7:42 AM

Change 534760 had a related patch set uploaded (by Phedenskog; owner: Phedenskog):
[performance/synthetic-monitoring-tests@master] Move Browsertime Chrome setup to Browsertime section.

https://gerrit.wikimedia.org/r/534760

gerritbot added a comment.Sep 6 2019, 3:10 PM

Change 534760 merged by jenkins-bot:
[performance/synthetic-monitoring-tests@master] Move Browsertime Chrome setup to Browsertime section.

https://gerrit.wikimedia.org/r/534760

• Peter added a subtask: T231870: Add new Graphite instance in Grafana.Sep 9 2019, 6:42 AM
gerritbot added a comment.Sep 9 2019, 6:49 AM

Change 535012 had a related patch set uploaded (by Phedenskog; owner: Phedenskog):
[performance/synthetic-monitoring-tests@master] Use the default WebPageTest agent.

https://gerrit.wikimedia.org/r/535012

gerritbot added a comment.Sep 9 2019, 7:19 AM

Change 535012 merged by jenkins-bot:
[performance/synthetic-monitoring-tests@master] Use the default WebPageTest agent.

https://gerrit.wikimedia.org/r/535012

gerritbot added a comment.Sep 9 2019, 7:27 AM

Change 535118 had a related patch set uploaded (by Phedenskog; owner: Phedenskog):
[performance/synthetic-monitoring-tests@master] Split WebPageTest configuration in one per browser.

https://gerrit.wikimedia.org/r/535118

gerritbot added a comment.Sep 9 2019, 8:07 AM

Change 535120 had a related patch set uploaded (by Phedenskog; owner: Phedenskog):
[performance/synthetic-monitoring-tests@master] Make it possible to override configurations per test.

https://gerrit.wikimedia.org/r/535120

gerritbot added a comment.Sep 9 2019, 10:15 AM

Change 535118 merged by jenkins-bot:
[performance/synthetic-monitoring-tests@master] Split WebPageTest configuration in one per browser.

https://gerrit.wikimedia.org/r/535118

gerritbot added a comment.Sep 9 2019, 10:16 AM

Change 535120 merged by jenkins-bot:
[performance/synthetic-monitoring-tests@master] Make it possible to override configurations per test.

https://gerrit.wikimedia.org/r/535120

gerritbot added a comment.Sep 9 2019, 10:19 AM

Change 535140 had a related patch set uploaded (by Phedenskog; owner: Phedenskog):
[performance/synthetic-monitoring-tests@master] Decrease number of runs for Alexa tests to 1.

https://gerrit.wikimedia.org/r/535140

gerritbot added a comment.Sep 9 2019, 10:23 AM

Change 535141 had a related patch set uploaded (by Phedenskog; owner: Phedenskog):
[performance/synthetic-monitoring-tests@master] Prepare to run multiple tests after each other.

https://gerrit.wikimedia.org/r/535141

gerritbot added a comment.Sep 9 2019, 11:51 AM

Change 535141 merged by jenkins-bot:
[performance/synthetic-monitoring-tests@master] Prepare to run multiple tests after each other.

https://gerrit.wikimedia.org/r/535141

gerritbot added a comment.Sep 9 2019, 11:51 AM

Change 535140 merged by jenkins-bot:
[performance/synthetic-monitoring-tests@master] Decrease number of runs for Alexa tests to 1.

https://gerrit.wikimedia.org/r/535140

gerritbot added a comment.Sep 9 2019, 12:12 PM

Change 535168 had a related patch set uploaded (by Phedenskog; owner: Phedenskog):
[performance/synthetic-monitoring-tests@master] Run multiple tests after each other on the same machine.

https://gerrit.wikimedia.org/r/535168

gerritbot added a comment.Sep 9 2019, 12:31 PM

Change 535168 merged by jenkins-bot:
[performance/synthetic-monitoring-tests@master] Run multiple tests after each other on the same machine.

https://gerrit.wikimedia.org/r/535168

gerritbot added a comment.Sep 9 2019, 12:39 PM

Change 535176 had a related patch set uploaded (by Phedenskog; owner: Phedenskog):
[performance/synthetic-monitoring-tests@master] Correct variabe name for tests.

https://gerrit.wikimedia.org/r/535176

gerritbot added a comment.Sep 9 2019, 12:41 PM

Change 535177 had a related patch set uploaded (by Phedenskog; owner: Phedenskog):
[performance/synthetic-monitoring-tests@master] Config paths missed config dir (depends on where you start the script)

https://gerrit.wikimedia.org/r/535177

gerritbot added a comment.Sep 9 2019, 12:42 PM

Change 535176 merged by jenkins-bot:
[performance/synthetic-monitoring-tests@master] Correct variabe name for tests.

https://gerrit.wikimedia.org/r/535176

gerritbot added a comment.Sep 9 2019, 12:42 PM

Change 535177 merged by jenkins-bot:
[performance/synthetic-monitoring-tests@master] Config paths missed config dir (depends on where you start the script)

https://gerrit.wikimedia.org/r/535177

gerritbot added a comment.Sep 9 2019, 12:57 PM

Change 535179 had a related patch set uploaded (by Phedenskog; owner: Phedenskog):
[performance/synthetic-monitoring-tests@master] Use correct script name for finding configuration.

https://gerrit.wikimedia.org/r/535179

gerritbot added a comment.Sep 9 2019, 12:59 PM

Change 535179 merged by jenkins-bot:
[performance/synthetic-monitoring-tests@master] Use correct script name for finding configuration.

https://gerrit.wikimedia.org/r/535179

• Peter closed subtask T232467: WebPagTest key configuration doesn't work in three steps as Resolved.Sep 10 2019, 12:19 PM
gerritbot added a comment.Sep 11 2019, 12:25 PM

Change 534131 merged by jenkins-bot:
[performance/synthetic-monitoring-tests@master] Setup a readme, explain what we do.

https://gerrit.wikimedia.org/r/534131

gerritbot added a comment.Sep 11 2019, 12:55 PM

Change 535836 had a related patch set uploaded (by Phedenskog; owner: Phedenskog):
[performance/synthetic-monitoring-tests@master] Run job on lower prio than the current test server.

https://gerrit.wikimedia.org/r/535836

gerritbot added a comment.Sep 11 2019, 12:58 PM

Change 535836 merged by jenkins-bot:
[performance/synthetic-monitoring-tests@master] Run job on lower prio than the current test server.

https://gerrit.wikimedia.org/r/535836

gerritbot added a comment.Sep 12 2019, 6:27 AM

Change 535997 had a related patch set uploaded (by Phedenskog; owner: Phedenskog):
[performance/synthetic-monitoring-tests@master] Update to stabel release of sitespeed.io.

https://gerrit.wikimedia.org/r/535997

gerritbot added a comment.Sep 12 2019, 10:40 AM

Change 535997 merged by jenkins-bot:
[performance/synthetic-monitoring-tests@master] Update to stable release of sitespeed.io.

https://gerrit.wikimedia.org/r/535997

gerritbot added a comment.Sep 14 2019, 7:44 PM

Change 536744 had a related patch set uploaded (by Phedenskog; owner: Phedenskog):
[performance/synthetic-monitoring-tests@master] Bump sitespeed.io with a fix where the timer video do not flicker.

https://gerrit.wikimedia.org/r/536744

gerritbot added a comment.Sep 14 2019, 7:45 PM

Change 536744 merged by jenkins-bot:
[performance/synthetic-monitoring-tests@master] Bump sitespeed.io with a fix where the timer video do not flicker.

https://gerrit.wikimedia.org/r/536744

• Peter closed subtask T233799: WebPageTest missed out on Hero timings as Resolved.Sep 25 2019, 12:20 PM
• Peter closed subtask T233726: Collect and graph stdev for all timing metrics as Resolved.Sep 25 2019, 12:22 PM
gerritbot added a comment.Sep 28 2019, 8:48 AM

Change 539653 had a related patch set uploaded (by Phedenskog; owner: Phedenskog):
[performance/synthetic-monitoring-tests@master] Remove Docker volumes for each iteration.

https://gerrit.wikimedia.org/r/539653

gerritbot added a comment.Sep 28 2019, 10:33 AM

Change 539653 merged by jenkins-bot:
[performance/synthetic-monitoring-tests@master] Remove Docker volumes for each iteration.

https://gerrit.wikimedia.org/r/539653

• Peter closed subtask T234177: How to trigger WebPageTest runs as Resolved.Sep 30 2019, 11:10 AM
gerritbot added a comment.Oct 3 2019, 5:00 AM

Change 540529 had a related patch set uploaded (by Phedenskog; owner: Phedenskog):
[performance/synthetic-monitoring-tests@master] Disable videos.

https://gerrit.wikimedia.org/r/540529

gerritbot added a comment.Oct 3 2019, 5:54 AM

Change 540529 merged by jenkins-bot:
[performance/synthetic-monitoring-tests@master] Disable videos.

https://gerrit.wikimedia.org/r/540529

• Peter triaged this task as High priority.Oct 3 2019, 11:29 AM
Krinkle merged a task: T118133: Devise a proper configuration file format for WebPageTest jobs.Oct 5 2019, 12:35 AM
Krinkle added subscribers: ori, Krinkle, Aklapper.
gerritbot added a comment.Oct 15 2019, 10:09 AM

Change 543106 had a related patch set uploaded (by Phedenskog; owner: Phedenskog):
[performance/synthetic-monitoring-tests@master] Collect and store CSS/JS/HTML for Firefox tests.

https://gerrit.wikimedia.org/r/543106

gerritbot added a comment.Oct 15 2019, 6:01 PM

Change 543106 merged by jenkins-bot:
[performance/synthetic-monitoring-tests@master] Collect and store CSS/JS/HTML for Firefox tests.

https://gerrit.wikimedia.org/r/543106

gerritbot added a comment.Oct 16 2019, 1:09 PM

Change 543452 had a related patch set uploaded (by Phedenskog; owner: Phedenskog):
[performance/synthetic-monitoring-tests@master] Bump to 10.3.0.

https://gerrit.wikimedia.org/r/543452

gerritbot added a comment.Oct 16 2019, 1:54 PM

Change 543452 merged by jenkins-bot:
[performance/synthetic-monitoring-tests@master] Bump to 10.3.0.

https://gerrit.wikimedia.org/r/543452

CDanis closed subtask T231870: Add new Graphite instance in Grafana as Resolved.Oct 30 2019, 1:55 PM
• Peter closed subtask T234414: Measure user journeys as Resolved.Nov 6 2019, 11:45 AM
• Peter removed a subtask: T235604: Send errors to IRC for the new synthetic testing setup.
gerritbot added a comment.Nov 12 2019, 8:42 AM

Change 550421 had a related patch set uploaded (by Phedenskog; owner: Phedenskog):
[performance/synthetic-monitoring-tests@master] Remove low prio run for WebPageTest.

https://gerrit.wikimedia.org/r/550421

gerritbot added a comment.Nov 12 2019, 8:45 AM

Change 550421 merged by jenkins-bot:
[performance/synthetic-monitoring-tests@master] Remove low prio run for WebPageTest.

https://gerrit.wikimedia.org/r/550421

• Peter closed this task as Resolved.Nov 12 2019, 8:56 AM
• Peter closed subtask T237521: Change WebPageTest dashboards and alerts to the new setup as Resolved.

The old setup is stopped and we only use the new now.

• Peter mentioned this in rPSMT807fcefe437b: Unify all synthetic testing into one git repo..Apr 8 2024, 3:13 AM
• Peter mentioned this in rPSMT7a041cbe37b8: Remove URLs that block AWS traffic..
• Peter mentioned this in rPSMT737c6abefa3e: Remove background between page loads to fix early First Visual Change on videos..
• Peter mentioned this in rPSMTeba03113eb4f: Median wait/read time is 21 seconds so let us run with that..
• Peter mentioned this in rPSMT17327e9774df: Set alias of connectivity so we can see what actual alias is used..
• Peter mentioned this in rPSMT24853077015b: Renamed structure to not include dot since it broke the namespace handling..
• Peter mentioned this in rPSMT7841b36a067b: Add white/clean the HTML so that FirstVisualChange is ok for mobile logged in..
• Peter mentioned this in rPSMTa254c23f5abf: Generic hide all elements to avoid submit button to affect First Visual Change.
• Peter mentioned this in rPSMTafe7c8fe4ff5: Enable showing the scipt that runs the tests..
• Peter mentioned this in rPSMT83786fd91962: Also run emulated runs on WebPageTest.
• Peter mentioned this in rPSMTcd7650a9183f: Use new test location for WebPageTest agent when we are testing the setup..
• Peter mentioned this in rPSMTbb0d37d7b4f5: Add meta data to each script to make it to see what we are testing..
• Peter mentioned this in rPSMT7522625e70f9: Post test for WebPageTest under the webpagetest namespace..
• Peter mentioned this in rPSMTf66adc6888e7: Add Firefox tests..
• Peter mentioned this in rPSMTb0ed7e88643a: Remove disable Browsertime..
• Peter mentioned this in rPSMT83d7190c1a5b: Disable video for Browsertime to override Docker default video settings..
• Peter mentioned this in rPSMT420613770e0a: Use latest Docker container with fixes for annotations using WebPageTest..
• Peter mentioned this in rPSMTd186aa2aa75f: Move tests to multiple files to follow WebPageReplay standard..
• Peter mentioned this in rPSMT5b5ec7f0e6c7: Verify that the input directory for tests really exists..
• Peter mentioned this in rPSMTa3b5e48cbf59: Bump sitespeed.io version to fix connectivity alias bug..
• Peter mentioned this in rPSMT78a65cc3f8c6: Add missing mobile configuration that sets correct view port & user agent..
• Peter mentioned this in rPSMT276d74b6174b: Mobile configuration was at the wrong place..
Krinkle mentioned this in rPSMT89d13fd551f3: Move Browsertime Chrome setup to Browsertime section..Apr 8 2024, 3:13 AM
• Peter mentioned this in rPSMTb31703b96da4: Remove --env use in Docker to simplify setup..Apr 8 2024, 3:13 AM
• Peter mentioned this in rPSMTe9b5552797a2: Use the default WebPageTest agent..
• Peter mentioned this in rPSMT93df9d251555: Make it possible to override configurations per test..
• Peter mentioned this in rPSMT23a44cda7f05: Split WebPageTest configuration in one per browser..
• Peter mentioned this in rPSMT5fad5f999dd5: Prepare to run multiple tests after each other..
• Peter mentioned this in rPSMT6a2545505e17: Decrease number of runs for Alexa tests to 1..
• Peter mentioned this in rPSMT679f65c06fcb: Run multiple tests after each other on the same machine..
• Peter mentioned this in rPSMT3b9abdc2abb1: Correct variabe name for tests..
• Peter mentioned this in rPSMTfe3bbec09dfd: Config paths missed config dir (depends on where you start the script).
• Peter mentioned this in rPSMTc9831de7b9fe: Use correct script name for finding configuration..
• Peter mentioned this in rPSMTc8124ae388e2: Setup a readme, explain what we do..
• Peter mentioned this in rPSMT7bd4c8d86d6d: Run job on lower prio than the current test server..
• Peter mentioned this in rPSMTff1cab5d0387: Update to stable release of sitespeed.io..
• Peter mentioned this in rPSMT6abd2cfc4a05: Bump sitespeed.io with a fix where the timer video do not flicker..
• Peter mentioned this in rPSMT78f538eb6fbf: Remove Docker volumes for each iteration..
• Peter mentioned this in rPSMT431c63f20ac9: Disable third party plugin..
• Peter mentioned this in rPSMTa3f585c0edc8: Disable videos..
• Peter mentioned this in rPSMT37381b244c5b: Collect and store CSS/JS/HTML for Firefox tests..
• Peter mentioned this in rPSMTccf2fca037e9: Bump to 10.3.0..
• Peter mentioned this in rPSMT32e9e22f80d6: Remove low prio run for WebPageTest..
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits