Right now, Parsoid's roundtrip tests run on ruthenium.eqiad.wmnet. As part of the test runs, we also collect performance metrics (parse latencies, serialization latencies, etc.). However, we cannot reliably use these metrics right now because the load on ruthenium varies quite a lot depending on what else is running on it.
Right now, it runs the following:
- roundtrip testing clients (which is roughly equivalent to what happens in production)
- the roundtrip testreduce server (which doles out test jobs to the rt-testing clients)
- mysql server (hosts the testreduce database)
- visual diff testing testreduce server
- visual diff testing clients which then spawn phantomjs processes (which once in a while get stuck)
- on occasion, @GWicke runs some dumper scripts
On top of this, our testreduce server code that provides the web interface for our roundtrip test results and visual diff test results has mysql queries that haven't been tuned for a long time and once in a while they load the mysql server.
On occasion, when all the above are running at the same time, swap space is zero or close to zero.
To make a long story short, we are requesting a dedicated spare server on which we run nothing else but the roundtrip testing clients which let us isolate parsing and serialization requests without having them compete with everything else. All the other things above can continue to run on ruthenium. This will now let us use the performance metrics gathered during roundtrip tests as a semi-reliable indicator of how our code might change performance in production when said code is deployed there.