@yuvipanda, @bd808, @Krenair hey, I have followed instructions from https://wikitech.wikimedia.org/wiki/Help:MediaWiki-Vagrant_in_Labs, but when I run vagrant up in step 7, I get the following message:

No usable default provider could be found for your system.

Vagrant relies on interactions with 3rd party systems, known as
"providers", to provide Vagrant with resources to run development
environments. Examples are VirtualBox, VMware, Hyper-V.

The easiest solution to this message is to install VirtualBox, which
is available for free on all major platforms.

If you believe you already have a provider available, make sure it
is properly installed and configured. You can see more details about
why a particular provider isn't working by forcing usage with
`vagrant up --provider=PROVIDER`, which should give you a more specific
error message for that particular provider.

What should I do to proceed? Thanks

You should have access to http://perf-testing.wmflabs.org/ and be able to use that although right now I'm having 502 Bad Gateway errors.

I'm also getting 502 Bad Gateway.

I actually have no idea - haven't used vagrant in years :( I am not able to
help, sorry. Maybe ask @bd808?

Baha - I've set this up on labs:

http://perf-testing.wmflabs.org/wiki/Pablo_Escobar
http://perf-testing.wmflabs.org/wiki/Pablo_Escobar2
http://perf-testing.wmflabs.org/wiki/Hannah_Montana
http://perf-testing.wmflabs.org/wiki/Hannah_Montana2
http://perf-testing.wmflabs.org/wiki/Eazy-E
http://perf-testing.wmflabs.org/wiki/Eazy-E2
http://perf-testing.wmflabs.org/wiki/Serena_Williams
http://perf-testing.wmflabs.org/wiki/Serena_Williams2
http://perf-testing.wmflabs.org/wiki/Ice_Cube
http://perf-testing.wmflabs.org/wiki/Ice_Cube2

There's a few rendering errors I'm looking to fix but they shouldn't impact your testing @bmansurov so feel free to go ahead on the above pages.

@yuvipanda, @Jdlrobson, thanks!

Sync up with @Peter so we can make this activity as easy as possible going forward

The activity couldn't be made any simpler imo. Tests ran smoothly and fast (about a couple minutes each).

You seem to have used webpagetest.org not the performance team's tool - https://github.com/wikimedia/wpt-reporter
How did the table get generated? Did you write your own script? We need to get to a point anyone can fire off a test on some pages.

Being able to use this is better as it allows us in future to run tests on larger sets of pages.
Sorry I should have been clearer in the request.

In meantime I'll crunch the data you've generated and see if we can learn anything.

@Jdlrobson, @Peter how do I get an API key for http://wpt.wmftest.org/? According to http://wpt.wmftest.org/getkey.php I can't do it automatically. Should I create a ticket or ping someone on IRC?

These results look great baha. Seems we can certainly count on labs for these tests. Time to first byte are pretty consistent and all show a radical improvement as we'd expect.

I'd be keen to run this on a larger sample next sprint if we can do that easily.

Results

• Cable (5/1 Mbps 28ms RTT). Ignore the connectivity column (2G) in the table below.
• Using https://github.com/wikimedia/wpt-reporter

Notes

1. us-east-1 is used by Jenkins, so tests were carried out on us-west-1 in order not to compete with Jenkins for resources.
2. Amazon EC2 takes about 10 mins to fire up an instance on us-west-1. If the instance is not used for more than an hour, it will be shut down.
3. 10 pages, at 9 runs each, took about 30 mins to run.
4. Chrome is used because Firefox reports wrong time to first byte.

How to

1. reading-web-performance-test.reading-web-staging.eqiad.wmflabs is set up for carrying out such tests.
2. The wpt-reporter tool hadn't been published to NPM at the time when the tests were carried out, so clone it from https://github.com/wikimedia/performance-WebPageTest
3. Copy
   test-urls.txt3 KBDownload
   to the server mentioned in step 2. Optionally add more pages or update the existing ones.
4. Get an API key from Peter (phedenskog in #wikimedia-mobile)
5. Set some environment variables:

export WPT_RUNS=9
export WPT_CONNECTIVITY=2G
export WMF_WPT_KEY=<THE-KEY-YOU-GOT-FROM-PETER>

6. Run tests: node wpt-reporter/bin/index.js --batch test-urls.txt
7. result.csv will be output once tests finish running.

Hey @bmansurov would be great to update https://www.mediawiki.org/wiki/Reading/Web/Performance/How_to_test_static_pages#On_Wikimedia.27s_WebPageTest_instance

I'm glad this was a success for you :)

@bmansurov I'm not convinced you ran your tests on a 2G connection given first view was under 1s. It might be that the tests were run on 3G. Can you print the test urls used and investigate that, if necessary?

Also I would recommend that we update the script to include all the job urls in the output table to allow people to review them in detail if they choose (if I had those now I'd be able to check if it was 2G or not) - cc @Peter

Assuming this is 3G, and using your previous data I compiled this:
https://docs.google.com/spreadsheets/d/1D-sfeRs8c8Dq6f540ea5vU4Aik6b1bKP9FI8Ukh__og/edit?usp=sharing

I documented my analysis here:
https://www.mediawiki.org/w/index.php?title=Reading%2FWeb%2FProjects%2FBarack_Obama_in_under_15_seconds_on_2G&type=revision&diff=1933639&oldid=1930793

In T115073#1786883, @Jdlrobson wrote:

@bmansurov I'm not convinced you ran your tests on a 2G connection given first view was under 1s. It might be that the tests were run on 3G. Can you print the test urls used and investigate that, if necessary?

That's weird. The table above (which is output by the script) says the connection was a 2G one. Yet, the test url (http://wpt.wmftest.org/result/151105_45_4Q/) says it was a cable connection. Following https://wikitech.wikimedia.org/wiki/WebPageTest#Test_with_2g_connectivity now...

Results

• Mobile 2G - Slow (35 Kbps/32 Kbps 1300ms RTT)
• It took about an hour to test all 10 pages (9 runs each)
• test-urls.txt3 KBDownload

In T115073#1786926, @Jdlrobson wrote:

Also I would recommend that we update the script to include all the job urls in the output table to allow people to review them in detail if they choose

Is this done? I will sign off when this is done as I cannot confirm myself that this is a 2G connection.
Thanks for running through this Baha.

^ @bmansurov

@Jdlrobson, yes, that's already done. See the test-urls.txt file in the earlier posts.

As @bmansurov states, that the tests were run with a 2G connection can be verified by looking at the test-urls.txt file.

Edit

… and I did, obviously.