https://grafana.wikimedia.org/dashboard/db/webpagetest?panelId=20&fullscreen&orgId=1&from=1489536000000&to=1491163200000

Seems the change happened between 2017-03-23 20:45 and 2017-03-24 03:00. Here a some of the deployments logged between Thursday 2017-03-23 12:00 and Friday 2017-03-24 03:00:

https://tools.wmflabs.org/sal/production?p=0&q=&d=2017-03-23

• 2017-03-23 19:11 <thcipriani@tin> rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis to 1.29.0-wmf.17
• 2017-03-23 20:16 <thcipriani@tin> rebuilt wikiversions.php and synchronized wikiversions files: all wikis to 1.29.0-wmf.17
• 2017-03-23 23:12 <thcipriani@tin> Synchronized php-1.29.0-wmf.17/extensions/ORES: SWAT: [[gerrit:344555|Stats: Invert "false" thresholds so they are correct]]

https://tools.wmflabs.org/sal/production?p=0&q=&d=2017-03-24

• 01:15 <catrope@tin> Started scap: Wikidata cherry-picks
• 01:40 <catrope@tin> Finished scap: Wikidata cherry-picks (with i18n)
• 02:15 <MaxSem> < Krinkle > (fixing outage)
• 02:25 <Krinkle> All apaches are back up

Seems like the obvious trigger is the roll out of this weeks' MediaWiki changes (1.29.0-wmf.17). Looking through the commits in MediaWiki core (not looking at the extensions yet), the following stand out in terms of frontend behaviour that may have changed for plain page views.

• ac817da014 - mediawiki.widgets: Remove autoEllipsis() from TitleOptionWidget (2 weeks ago) <Timo Tijhof>
• f4329ddbc8 - resourceloader: Use SVG url when ResourceLoaderImageModule can't embed (3 weeks ago) <WMDE-Fisch>
• 0638e8568f - Make infused and non-infused DateInputWidget consistent (4 weeks ago) <Geoffrey Mon>
• 692bf795a9 - Skin: Only load jquery.makeCollapsible if needed (4 weeks ago) <Timo Tijhof>
• 2fccd229d2 - Skin: Preload jquery.tablesorter based on rough heuristics (4 weeks ago) <Timo Tijhof>

Next step: Compare waterfall collected by WebPageTest from runs before/after the change.

PS: Navigation Timing from around the same time does not seem to have significantly changed so it might be specific to Firefox or certain aspects of this run (Facebook page, English Wikipedia, cable speed, geolocation). (See https://grafana.wikimedia.org/dashboard/db/navigation-timing?from=1489795200000&to=1490572800000, switch between the various metrics).

Runs before the change alternate between SpeedIndex 2500 and 2800 on the graph. A few samples:

Runs after the change alternate between SpeedIndex 2200 and 2500 on the graph. A few samples:

The removed request was done in rMW2fccd229d2e8: Skin: Preload jquery.tablesorter based on rough heuristics (part of T159911: Audit modules (2017)). I tried hard to find some other difference but this seems to be the only thing that stands out.

The earlier loading of tablesorter explains:

• Visually Complete being reached earlier since the loading of tablesorter adds little arrow icons inside some of the tables on the page. This process now starts earlier instead being blocked on a lazily discovered http request.
• "Load Time" (domComplete/window.load) happening earlier because the lazy load for "tablesorter" used to happen inside a document ready handler (domInteractive). And any requests that start before domContentLoadedEventEnd will count as subresources and thus hold back the domComplete metric. For the same reason, the background request for login.wikimedia.org (as example) is intentionally delayed beyond document ready so that it doesn't cause the page load indicator in native browsers to be prolonged.
• "Bytes In (domComplete)" being slightly larger (+4KB) as it now includes tablesorter.
• "Bytes In (fullyLoaded)" being slightly smaller (-2KB, which includes the same 4KB still). Presumably because we no longer have the added overhead of another module request. It might also be that the code compresses slightly better now.

However none of this explains why it affected earlier metrics like "Start Render". Browsers don't have to wait for document ready (domInteractive) or window load (domComplete) to start painting something.

On the other side, the request that was removed used to start around 1.9s, and start render at 2.3s. So it certainly is "in time" to have been used as a heuristic of some sort in Firefox internally. While Firefox definitely doesn't wait for domComplete to render, it might be that it more-or-less arbitrarily decides to wait a small amount of time for certain requests to complete in case they significantly change the page (to avoid FOUC). It looks like it was doing that for our tablesorter request which Firefox may've seen as "important" for some reason.