Page MenuHomePhabricator

Get typical upper module build time under 1 second
Closed, ResolvedPublic

Description

From https://grafana.wikimedia.org/dashboard/db/resourceloader

Screen Shot 2016-07-29 at 20.36.15.png (924×2 px, 239 KB)

  • p99 and upper: 200ms (min in last 24h), 1min (max in last 24h)

This means that every day, every minute, at least one module build takes 200ms. Note that we often a dozen or more modules in a single HTTP request. Not to mention the startup module, which has to compute many more (though for the common case of file modules we have optimisations in place that bypass the module build).

The highest point is even over a minute.

Screen Shot 2016-07-29 at 20.36.29.png (928×2 px, 384 KB)

  • p95: 1.9ms (highest point in last 3 months)

On the other hand, our p95 has not gone above 2ms in over 3 months. This is quite an accomplishment and largely due to our content-based caching for minification and Less compilation. There are many different urls to the same resource essentially (N wikis * N user languages * N skins), and most of these don't vary, so once computed, they're re-used until the resource changes.

However this hasn't changed the fact that Less compilation and other code involved in module build still seems pretty darn slow in case of cache miss.

Screen Shot 2016-07-29 at 20.36.43.png (1×1 px, 185 KB)

Over a 24 hour period:

modulemaxavg
ext_cx_header5.3s949.3ms
ext_guidedTour_styles4.2s2.4s
mobile_editor_ve2.2s1.5s
mobile_notifications_overlay2.2s1.4s
skins_minerva_special_watchlist_styles2.0s1.1s
mobile_special_uploads_styles1.9s990.9ms
mobile_backtotop1.9s952.1ms
mobile_app_pagestyles_android1.8s911.2ms
ext_flow_visualEditor1.7s1.2s
ext_wikimediaBadges1.4s1.0s

Plenty of modules that are consistently slow (not just peaks).

From: https://grafana.wikimedia.org/dashboard/db/resourceloadermodule

Graph: Build times for skins_vector_styles

Screen Shot 2016-07-29 at 20.53.06.png (716×2 px, 265 KB)

Event Timeline

Krinkle renamed this task from Get upper module build time under 1 second to Get typical upper module build time under 1 second.Jul 29 2016, 7:55 PM
Krinkle triaged this task as Medium priority.Aug 1 2016, 8:02 PM
Krinkle moved this task from Inbox to Backlog on the MediaWiki-ResourceLoader board.
Krinkle moved this task from Inbox to Backlog: Maintenance on the Performance-Team board.
Krinkle lowered the priority of this task from Medium to Low.

Closing in favour of T164248: HTTP responses from app servers sometimes stall for >1s.

Build times have improved by 20-60% over the past year due to numerous improvements in different areas. But the spikes still exist, albeit with lower frequency nowadays.

https://grafana.wikimedia.org/dashboard/db/resourceloader?panelId=26&fullscreen&refresh=5m&orgId=1

Screen Shot 2017-05-02 at 20.32.26.png (934×2 px, 280 KB)