Work in progress :)

Background

We want to know what kind of impact lazy loading images would have from a performance perspective. My first feeling was that the fully loaded metric (the fully loaded happens when all responses has been downloaded to the browser) should be much smaller if we downloads less images. But when we tested serving the Obama page on a 2G connection and a special version with much less image data (1 mb), the numbers are almost the same (we use SPDY). However during these tests we had some problems with the login requests (redirects and 404), so that could have had impacts.

The test

Lets test how it really works. @ori has prepared a couple of different static versions of the mobile version of the Obama page. Lets also test what can give us most impact, we have a couple of different pages:

• One original page, the way it works today, the base to compare with
• One using the NetSpeed B cookie, making us serve 1 mb less image data to the user
• One without images (<img src=..> rewritten to <img data-src=..>) - this will simulate lazy loading images.
• One without the references section (decreasing the amount of HTML)
• One without the top CSS not inlined, just without it

We test the pages using WebPageTest. We will do 5 runs for each page and take the median run. We will test on Chrome desktop (emulate mobile, set user agent to Iphone 6) and test on Chrome on a Moto G so that we also tests on a real device.

The metrics we collect:

• Start render - when something first happens on the screen.
• Speed Index - simplified it explains when the content within the current viewport is ready for the user. Smaller values are better.
• Fully loaded when everything has been downloaded to the browser.

Emulated mobile 2G Fast using SPDY

Do we get the same numbers testing on a real device?

Motorola G 2G Fast using SPDY

What do we see?

• Lazy loading images will make the fully loaded happen much earlier. That's really good. It will not give any extras for start rendering or SpeedIndex (or only a small small difference).
• To really make a difference for start rendering and SpeedIndex we need to serve less HTML and inline the CSS.
• The start rendering between emulating a mobile browser using Chrome and a real device doesn't have a so big difference in this case. That's good to know.

However, there's a elephant in the room :) If you check the waterfall graphs you will see that when we do request a CSS file, it takes long time to actually get it (13 seconds). This example is from the original page:

Since we are doing a couple of requests/responses at the same time, it is not strange that the download time (the blue part) is longer, because we are on a limited connection. But we also have a long response time for the server (green). The time to first byte is 7614 ms and download time is 5601 ms.

When I test using same browser/same throttling but forces to use HTTP/1 it looks like this:

The time to first byte for the CSS is 1401 ms and download time is 356 ms. The download time could maybe be explained that we only do the CSS download at that time, but the first byte I don't know. @ori @Krinkle do you have an idea?

Lets test what it looks like using HTTP/1.

SPDY vs HTTP/1

Lets check what the numbers looks like when we use HTTP/1.1. We can configure Chrome to use HTTP/1 by setting the flag --use-spdy=false. Then the browser will do only X requests at a time to each domain. The fully loaded time will probably be longer. Lets measure it to know:

Emulated mobile 2G Fast using HTTP/1

It looks like serving with HTTP/1 we already got almost a reasonable start rendering time. Lets investigate the TTFB before we continue.

Do we test the "right" way

Could it be that I do the testing wrongly somehow? Please give feedback if see something that looks strange.