What version of Chrome is actually used in the agents? When we keep an agent up and running, it is configured to download the latest version of the agent. But if we just start one instance run a couple of tests, and then closes it, it will use the version in the AWS image.
Running on Jenkins this would be fine but if we want to test/evaluate changes for the mobile team (where we spin up an instance, test massive and then closes, this could have an impact.
Testing with Jon, I've seen there has been a difference with certificates (=different Chrome versions) on running on west-1 and webpagetest.org.