Page MenuHomePhabricator

[6 hours] Test a subset of pages with only a lead section [take 2]
Closed, ResolvedPublic

Description

T113649 didn't quite go to plan so let's try this again. Someone other than Jon should do this.

Various static pages have been generated here: https://phabricator.wikimedia.org/P2172 - 1 which is the normal article, 1 with only the lead section - for the following pages:
https://en.wikipedia.org/wiki/Pablo_Escobar
https://en.wikipedia.org/wiki/Eazy-E
https://en.wikipedia.org/wiki/Hannah_Montana
https://en.wikipedia.org/wiki/Serena_Williams
https://en.wikipedia.org/wiki/Ice_Cube

These pages should be hosted on labs in a mediawiki like environment, where first byte is consistent in all the results, and visiting the pages should not generate any 404s and the JS should load successfully (note in the pages on Jon's site there are various 404 errors)

Test the performance lead section vs full article (ignore no images view since this can be done relatively easily on production) and report back your findings.

Sync up with @Peter so we can make this activity as easy as possible going forward

Duration: 6hrs

Related Objects

StatusAssignedTask
DeclinedNone
ResolvedCatrope
ResolvedSbailey
Resolved GWicke
Resolvedssastry
ResolvedNone
ResolvedDbrant
ResolvedbearND
ResolvedMholloway
ResolvedNone
OpenNone
OpenNone
OpenNone
OpenNone
OpenNone
OpenNone
OpenNone
ResolvedJdforrester-WMF
DuplicateNone
ResolvedbearND
OpenNone
OpenNone
DeclinedNone
DuplicateNone
DeclinedNone
OpenNone
OpenNone
ResolvedArlolra
ResolvedMooeypoo
ResolvedCatrope
Resolved GWicke
OpenNone
Resolvedmarcoil
Resolvedmarcoil
Resolved GWicke
DuplicateNone
DeclinedNone
OpenNone
OpenNone
Resolvedbmansurov

Event Timeline

Jdlrobson raised the priority of this task from to Low.
Jdlrobson updated the task description. (Show Details)
Jdlrobson updated the task description. (Show Details)Oct 26 2015, 10:27 PM
KLans_WMF renamed this task from Test a subset of pages with only a lead section [take 2] to [6 hours] Test a subset of pages with only a lead section [take 2].Oct 27 2015, 4:07 PM
bmansurov added a subscriber: yuvipanda.EditedOct 29 2015, 12:48 PM

@yuvipanda, @bd808, @Krenair hey, I have followed instructions from https://wikitech.wikimedia.org/wiki/Help:MediaWiki-Vagrant_in_Labs, but when I run vagrant up in step 7, I get the following message:

No usable default provider could be found for your system.

Vagrant relies on interactions with 3rd party systems, known as
"providers", to provide Vagrant with resources to run development
environments. Examples are VirtualBox, VMware, Hyper-V.

The easiest solution to this message is to install VirtualBox, which
is available for free on all major platforms.

If you believe you already have a provider available, make sure it
is properly installed and configured. You can see more details about
why a particular provider isn't working by forcing usage with
`vagrant up --provider=PROVIDER`, which should give you a more specific
error message for that particular provider.

What should I do to proceed? Thanks

You should have access to http://perf-testing.wmflabs.org/ and be able to use that although right now I'm having 502 Bad Gateway errors.

I'm also getting 502 Bad Gateway.

I actually have no idea - haven't used vagrant in years :( I am not able to
help, sorry. Maybe ask @bd808?

Document CompleteFully loaded
PageResultTypeLoad TimeFirst ByteStart RenderUser TimeSpeed IndexDOM ElementsTimeRequestsBytes InTimeRequestsBytes InCost
http://perf-testing.wmflabs.org/wiki/Pablo_Escobarhttp://www.webpagetest.org/result/151103_XC_JQH/normal157.416s4.876s13.287s157.355s158951334157.416s13558 KB161.236s14562 KB$$---
http://perf-testing.wmflabs.org/wiki/Pablo_Escobar2http://www.webpagetest.org/result/151103_XT_J6W/lead only95.180s5.224s9.093s95.097s1317223295.180s10310 KB98.973s11314 KB$----
http://perf-testing.wmflabs.org/wiki/Hannah_Montanahttp://www.webpagetest.org/result/151103_1N_KS6/normal148.021s4.992s13.688s147.857s150562995148.021s17523 KB152.012s18527 KB$$---
http://perf-testing.wmflabs.org/wiki/Hannah_Montana2http://www.webpagetest.org/result/151103_WT_JJ3/lead only92.789s5.071s8.991s92.620s1329627692.789s8310 KB96.622s9314 KB$----
http://perf-testing.wmflabs.org/wiki/Eazy-Ehttp://www.webpagetest.org/result/151103_6H_KSN/normal152.665s4.422s12.491s152.488s621992005152.665s14523 KB155.999s16527 KB$$---
http://perf-testing.wmflabs.org/wiki/Eazy-E2http://www.webpagetest.org/result/151103_8T_JK3/lead only120.100s5.340s10.589s120.052s50673279120.100s8418 KB123.872s9422 KB$----
http://perf-testing.wmflabs.org/wiki/Serena_Williamshttp://www.webpagetest.org/result/151103_46_KSY/normal277.130s4.987s13.590s242.813s1722712425277.130s911,009 KB279.568s921,013 KB$$$--
http://perf-testing.wmflabs.org/wiki/Serena_Williams2http://www.webpagetest.org/result/151103_68_JND/lead only123.207s4.953s8.690s123.151s14833463123.207s13438 KB126.957s14442 KB$----
http://perf-testing.wmflabs.org/wiki/Ice_Cubehttp://www.webpagetest.org/result/151103_85_KZC/normal152.352s5.063s13.390s152.198s408142067152.352s14532 KB156.311s15536 KB$$---
http://perf-testing.wmflabs.org/wiki/Ice_Cube2http://www.webpagetest.org/result/151103_MN_M29/lead only118.552s4.868s9.987s118.387s37537204118.552s9409 KB131.984s10413 KB$----

Sync up with @Peter so we can make this activity as easy as possible going forward

The activity couldn't be made any simpler imo. Tests ran smoothly and fast (about a couple minutes each).

You seem to have used webpagetest.org not the performance team's tool - https://github.com/wikimedia/wpt-reporter
How did the table get generated? Did you write your own script? We need to get to a point anyone can fire off a test on some pages.

Being able to use this is better as it allows us in future to run tests on larger sets of pages.
Sorry I should have been clearer in the request.

In meantime I'll crunch the data you've generated and see if we can learn anything.

@Jdlrobson, @Peter how do I get an API key for http://wpt.wmftest.org/? According to http://wpt.wmftest.org/getkey.php I can't do it automatically. Should I create a ticket or ping someone on IRC?

Florian removed a subscriber: Florian.Nov 4 2015, 11:07 AM

These results look great baha. Seems we can certainly count on labs for these tests. Time to first byte are pretty consistent and all show a radical improvement as we'd expect.

I'd be keen to run this on a larger sample next sprint if we can do that easily.

bmansurov added a comment.EditedNov 5 2015, 12:57 PM

Results

urllocationconnectivityfirstView.SpeedIndexfirstView.renderfirstView.TTFBfirstView.fullyLoadedfirstView.mwLoadStartfirstView.mwLoadEndfirstView.html.requestsfirstView.js.requestsfirstView.css.requestsfirstView.image.requestsfirstView.flash.requestsfirstView.font.requestsfirstView.other.requestsfirstView.html.bytesfirstView.js.bytesfirstView.css.bytesfirstView.image.bytesfirstView.flash.bytesfirstView.font.bytesfirstView.other.bytes
http://perf-testing.wmflabs.org/wiki/Pablo_Escobar?mobileaction=toggle_view_mobileus-west-1:Chrome2G22251684624362615003038162300032478153916786238658000
http://perf-testing.wmflabs.org/wiki/Pablo_Escobar2?mobileaction=toggle_view_mobileus-west-1:Chrome2G141813855823183122426931620000633315346578620000
http://perf-testing.wmflabs.org/wiki/Hannah_Montana?mobileaction=toggle_view_mobileus-west-1:Chrome2G18181779499354216672981162800038667153918786250376000
http://perf-testing.wmflabs.org/wiki/Hannah_Montana2?mobileaction=toggle_view_mobileus-west-1:Chrome2G141813725483175116225851620000648115346578620000
http://perf-testing.wmflabs.org/wiki/Eazy-E?mobileaction=toggle_view_mobileus-west-1:Chrome2G185417706143535162329831627000296691539187860130410000
http://perf-testing.wmflabs.org/wiki/Eazy-E2?mobileaction=toggle_view_mobileus-west-1:Chrome2G14951397455309212732616162100071031534637862101280000
http://perf-testing.wmflabs.org/wiki/Serena_Williams?mobileaction=toggle_view_mobileus-west-1:Chrome2G38882971589484425094081162820001396171539177861299619000
http://perf-testing.wmflabs.org/wiki/Serena_Williams2?mobileaction=toggle_view_mobileus-west-1:Chrome2G16341574559330013292685162300011821153465786240064000
http://perf-testing.wmflabs.org/wiki/Ice_Cube?mobileaction=toggle_view_mobileus-west-1:Chrome2G17891688530333515222772162700037311153917786292027000
http://perf-testing.wmflabs.org/wiki/Ice_Cube2?mobileaction=toggle_view_mobileus-west-1:Chrome2G1314128447027631049231216210006606153465786232681000

Notes

  1. us-east-1 is used by Jenkins, so tests were carried out on us-west-1 in order not to compete with Jenkins for resources.
  2. Amazon EC2 takes about 10 mins to fire up an instance on us-west-1. If the instance is not used for more than an hour, it will be shut down.
  3. 10 pages, at 9 runs each, took about 30 mins to run.
  4. Chrome is used because Firefox reports wrong time to first byte.

How to

  1. reading-web-performance-test.reading-web-staging.eqiad.wmflabs is set up for carrying out such tests.
  2. The wpt-reporter tool hadn't been published to NPM at the time when the tests were carried out, so clone it from https://github.com/wikimedia/performance-WebPageTest
  3. Copy to the server mentioned in step 2. Optionally add more pages or update the existing ones.
  4. Get an API key from Peter (phedenskog in #wikimedia-mobile)
  5. Set some environment variables:
export WPT_RUNS=9
export WPT_CONNECTIVITY=2G
export WMF_WPT_KEY=<THE-KEY-YOU-GOT-FROM-PETER>
  1. Run tests: node wpt-reporter/bin/index.js --batch test-urls.txt
  2. result.csv will be output once tests finish running.
Jdlrobson added a comment.EditedNov 5 2015, 9:51 PM

@bmansurov I'm not convinced you ran your tests on a 2G connection given first view was under 1s. It might be that the tests were run on 3G. Can you print the test urls used and investigate that, if necessary?

Also I would recommend that we update the script to include all the job urls in the output table to allow people to review them in detail if they choose (if I had those now I'd be able to check if it was 2G or not) - cc @Peter

bmansurov added a comment.EditedNov 6 2015, 1:55 AM

@bmansurov I'm not convinced you ran your tests on a 2G connection given first view was under 1s. It might be that the tests were run on 3G. Can you print the test urls used and investigate that, if necessary?

That's weird. The table above (which is output by the script) says the connection was a 2G one. Yet, the test url (http://wpt.wmftest.org/result/151105_45_4Q/) says it was a cable connection. Following https://wikitech.wikimedia.org/wiki/WebPageTest#Test_with_2g_connectivity now...

Results

  • Mobile 2G - Slow (35 Kbps/32 Kbps 1300ms RTT)
  • It took about an hour to test all 10 pages (9 runs each)
urllocationconnectivityfirstView.SpeedIndexfirstView.renderfirstView.TTFBfirstView.fullyLoadedfirstView.mwLoadStartfirstView.mwLoadEndfirstView.html.requestsfirstView.js.requestsfirstView.css.requestsfirstView.image.requestsfirstView.flash.requestsfirstView.font.requestsfirstView.other.requestsfirstView.html.bytesfirstView.js.bytesfirstView.css.bytesfirstView.image.bytesfirstView.flash.bytesfirstView.font.bytesfirstView.other.bytes
http://perf-testing.wmflabs.org/wiki/Pablo_Escobarus-west-1:Chromecustom784659874264178345950163361523000323651521177862215083000
http://perf-testing.wmflabs.org/wiki/Pablo_Escobar2us-west-1:Chromecustom658463794380168216217150991520000622215166478610000
http://perf-testing.wmflabs.org/wiki/Hannah_Montanaus-west-1:Chromecustom788060854301180336086160821528000385521521157862201373000
http://perf-testing.wmflabs.org/wiki/Hannah_Montana2us-west-1:Chromecustom648662834316169806214152491520000637115166478620000
http://perf-testing.wmflabs.org/wiki/Eazy-Eus-west-1:Chromecustom1097061844374180276290161831527000295541521147862185699000
http://perf-testing.wmflabs.org/wiki/Eazy-E2us-west-1:Chromecustom103866385433316970644615252152100069951516647862101280000
http://perf-testing.wmflabs.org/wiki/Serena_Williamsus-west-1:Chromecustom69126595449432647647831104152820001395111521177862833557000
http://perf-testing.wmflabs.org/wiki/Serena_Williams2us-west-1:Chromecustom680664934477180836399163371523000117081516647862127417000
http://perf-testing.wmflabs.org/wiki/Ice_Cubeus-west-1:Chromecustom990060864435180106482161751527000371981521177862138983000
http://perf-testing.wmflabs.org/wiki/Ice_Cube2us-west-1:Chromecustom10818639043021697163911524715210006499151662786292468000

Also I would recommend that we update the script to include all the job urls in the output table to allow people to review them in detail if they choose

Is this done? I will sign off when this is done as I cannot confirm myself that this is a 2G connection.
Thanks for running through this Baha.

@Jdlrobson, yes, that's already done. See the test-urls.txt file in the earlier posts.

phuedx added a subscriber: phuedx.

As @bmansurov states, that the tests were run with a 2G connection can be verified by looking at the test-urls.txt file.

Edit

… and I did, obviously.

phuedx closed this task as Resolved.Nov 10 2015, 12:36 PM

AFAICT all AC have been satisfied.