Move to bare metal servers for performance tests
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	Peter
	May 3 2022, 11:37 AM

Description

In T203060 we could see that bare metal servers for hosting would give us more stable metrics over time. I would like us to think about how many servers and where we want to have them so we can check for the budget. When I did the test I used a Swedish company called Glesys and the server spec where:

mobo: x9scd-f
cpu: e3-1230
ram: 1x8gb ddr3
storage: 1x 480gb ssd
8 CPUs

One thing that is important is that we need to run on Ubuntu (not Debian) so we can use Chrome instead of Chromium for our tests.

And I think should run on something similar.

Today on AWS we have:
1 server running Graphite (and I think we could keep that)
3 servers running WebPagReplay tests and user journey tests using sitespeed.io direct against Wikipedia
1 server running a WebPageTest agent
1 small server running a WebPageTest server

For WebPageReplay servers I was thinking we could have in our data centre but I'm not 100% sure since I guess we need formalise the containers we run? Right now we just use the current I release with sitespeed.io, meaning there's no overhead building new containers.

WebPageTest I'm thinking we should close it down but we should discuss it see T302279.

Lest all try sync what and how we want to run these tests the next year.

Related Objects
Search...

Status	Assigned	Task
Resolved	Peter	T307446 Move to bare metal servers for performance tests
Resolved	Peter	T203060 Evaluate deploying synthetic testing on bare metal
Resolved	Peter	T303182 WebPageReplay tests gets "Error: module already implemented"
Resolved	Peter	T303208 Investigate why the Firefox WebPageReplay for some URLs are worse on the bare metal server
In Progress	Peter	T304281 Write blog post about testing on bare metal
Resolved	Peter	T304282 Cleanup graphite data from the bare metal tests
Resolved	Peter	T311980 Move tests from AWS to bare metal
Resolved	Peter	T311981 Set up bare metal server at Hetzner for performance tests
Resolved	Peter	T311983 Setup tests on the new bare metal machine
Resolved	Peter	T311984 Remove AWS tests
Resolved	Peter	T311985 Switch alerts to use bare metal servers metrics
Resolved	Peter	T333524 Pin FFMPEG to specific CPU:s
Resolved	Peter	T312203 Test running synthetic performance test at hetzner
Resolved	Peter	T314752 Report number of running processes before and after tests
Resolved	larissagaulia	T317392 Bare metal at supplier.io or in-house
Resolved	Peter	T326710 Firefox have much higher variance in metrics

Event Timeline

Peter created this task.May 3 2022, 11:37 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 3 2022, 11:37 AM

I think what I need to know if if we can have servers running in our dc on an isolated network, maybe you know this @dpifke ? Those servers needs to be able to connect to internet but we do not want them to be connected in any way with the rest of our servers.

We should talk through what we would hope to accomplish by not having them connected to the rest of the DC.

I think we want them puppetized and managed like the rest of the fleet. This would be a vast improvement over the current AWS setup, and require less effort on our part, by letting us leverage the existing infrastructure tooling.

We can talk with SRE about the network details, but I'm dubious that any "isolated network" setup would be any fewer hops between the test server and the front-ends, or (even if it is an extra hop or two) that those hops would reduce the fidelity of the test results in any way.

Also, from a practical standpoint, I think we're much more likely to get approval to add these if we DON'T require a bunch of custom network setup and ongoing maintenance/complexity burden.

Peter moved this task from Inbox, needs triage to Doing: Goals on the Performance-Team board.May 9 2022, 6:07 PM

Peter moved this task from Doing: Goals to To-do: Goals prioritized current Quarter on the Performance-Team board.

Peter moved this task from To-do: Goals prioritized current Quarter to To-do: Goals, prioritized next 4 Quarters on the Performance-Team board.Oct 4 2022, 6:48 PM

Peter mentioned this in T317392: Bare metal at supplier.io or in-house.Oct 7 2022, 3:24 PM

Peter updated the task description. (Show Details)Oct 7 2022, 3:27 PM

Krinkle added a subtask: T203060: Evaluate deploying synthetic testing on bare metal.Oct 18 2022, 6:53 PM

Krinkle added a subtask: T311980: Move tests from AWS to bare metal.

MoritzMuehlenhoff subscribed.Nov 17 2022, 4:15 PM

LSobanski subscribed.Nov 17 2022, 5:02 PM

larissagaulia moved this task from To-do: Goals, prioritized next 4 Quarters to To-do: Goals prioritized current Quarter on the Performance-Team board.Jan 19 2023, 12:13 PM

Peter closed subtask T311980: Move tests from AWS to bare metal as Resolved.Apr 11 2023, 11:42 AM

Peter closed subtask T326710: Firefox have much higher variance in metrics as Resolved.